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ANALYZING COST, SCHEDULE, AND ENGINEERING 
VARIANCES ON ACQUISITION PROGRAMS 

ABSTRACT 


This study of cost, schedule, and engineering variance (CV, SV, and EV) data identified 
in the Selected Acquisition Reports (SARs) of acquisition programs indicates that early 
program variances are significantly associated with future program variances. An 
enhanced understanding of CV, SV, and EV interrelationships and the connection 
between these program variances and the cost and schedule Earned Value contract 
variances will allow program managers to better understand the full programmatic impact 
of a variance problem. This understanding could also aid future researchers in 
identifying best practices in recovering from the identification of such a problem. In 
addition, the identification of CV, SV, and EV differences across Major Defense 
Acquisition Program (MDAP) types highlights the connection between segments of the 
defense industry and the development of best program management practices. 

This research first examines data using traditional descriptive statistics in order to 
determine whether identifiable patterns exist among MDAPs and their associated 
contracts. 

A primary objective of the analysis is to develop empirical models that employ 
cross-sectional, time-series data contained in the SARs. These models help explain the 
full effect of fixed-price incentive R&D contracts within MDAPs on cost and schedule 
variance during both engineering and manufacturing development (EMD) and production 
and deployment. 

It is anticipated that this analysis will also help close any existing gaps in the 
understanding of program versus contract management data. 
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I. INTRODUCTION 


A. OVERVIEW 

The current policy climate in the federal government is one of increasing fiscal 
austerity. The Department of Defense (DoD) spent approximately $500 billion on 
contracted goods and services in fiscal year (FY) 2010, of which $135 billion was spent 
on procurement. Under Secretary of Defense for Acquisition, Technology, and Logistics 
(USD[AT&L]) Ashton Carter released a memo in September 2010 providing guidance 
for how the DoD could achieve an estimated $100 billion in management and contracting 
efficiencies over the course of five years (Carter, 2010). Decreasing budgets will require 
acquisition professionals to do more with less. 

GAO reported the FY2008 portfolio of 96 Major Defense Acquisition Programs 
(MDAPs) collectively ran $303 billion over initial program budget and were an average 
of 22 months behind schedule in delivering initial capability (GAO, 2011). The analysis 
of cost variance (CV), schedule variance (SV), and engineering variance (EV) differences 
across MDAPs could highlight defense acquisition industry segments with program 
management best practices. If not attributable to program management practices, 
differences across MDAPs could point to differing levels of technology insertion risk or 
system integration complexity. 

As acquisition professionals have attempted to increase efficiency and examine 
current contracting practices, one area of interest has been contract type. There has been 
a new emphasis within the acquisition community to increase the amount of fixed-price 
contracts. At one time, the DoD attempted to impose fixed-price incentive contracts on 
efforts in which significant invention could be anticipated, although recently the use of 
cost-plus-award-fee contracts has become widespread (Carter, 2010). Under Secretary 
Carter advocated the use of fixed-price incentive firm target contracts in the place of cost- 
plus-award-fee contracts wherever practicable (Carter, 2010). 

From Table 6.8 of the National Defense Budget Estimates for FY2012, also 
known as the DoD Greenbook, research and development (R&D) contracts for FY2010 
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totaled $80 billion (USD Comptroller, 2011). Increasing the number of R&D contracts 
issued on a fixed-price incentive basis could cause significant savings based on the 
presumption that fixed-price incentive contracts decrease cost overruns. During the 
engineering and manufacturing development (EMD) phase of programs, contracts funded 
through research, development, test, and evaluation (RDT&E) are of particular interest 
because they represent the last stage in which large cost-plus contracts are typically 
awarded. However, restricting R&D work to a fixed-price incentive basis could 
negatively affect the detection and solving of problems early in a program. 

Contracted companies may stop researching all problems and alternatives in order 
to increase their profit margin on the fixed-price incentive contracts, provided that the 
contract specification requirements are satisfied. Properly constructed incentives may 
mitigate the risks of contractors’ scope of research for a material solution, but 
quantitative incentives may not be able to motivate all the desired behaviors. 
Deficiencies in the identification of MDAP problems could increase costs in the long 
term. For example, if a contractor eliminated tests during RDT&E to save money for the 
company, then it could result in increased expenses to the government to fix the 
unidentified problems in the future. 

1. Impact of Acquisition Failures on the Warfighter and the Public 

Cost overruns and schedule delays have a significant impact on the warfighter and 
the taxpayers. Deputy Secretary of Defense William J. Lynn III stated, 

American taxpayers and our men and women in uniform are understandably 
skeptical when they hear promises to reform the Defense Department’s sprawling 
acquisition system, which often delivers major weapons systems to our troops 
years behind schedule and billions of dollars over budget. (Lynn, 2009) 

There have been numerous studies that have focused on improving defense acquisitions, 

but the DoD repeatedly experiences the same problems; cost and schedule growth have 

continued over the last 30 years regardless of all the acquisition reforms, congressional 

studies, and DoD reports repetitively highlighting the same issues (Schwartz, 2010). 

These include specific studies of successes (Dillard & Ford, 2009) and spectacular 

failures (General Accounting Office [GAO], 1992), as well as more systemic reforms 
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(Department of Defense Authorization Act, 1982; DoD, 2011). The aim of these efforts 
was to improve elements of the acquisition system, but they have not effectively 
addressed the reinforcing relationships present between fiscal constraints, state-of-the-art 
technical requirements, and the acute needs of the government, as explored in the 
discussion of risk by acquisition phase in Chapter II of this research. 

Program oversight generally addresses specific problems and not the process as a 
whole. Accordingly, it is extremely difficult to balance the pressures exerted by different 
reporting requirements over a program’s life cycle; it is possible that these reporting 
requirements, intended to improve acquisition outcomes, may decrease the probability of 
program success due to their onerous nature (Wood & Moseley, 2011). As a report from 
Business Executives for National Security (BENS, 2009) stated, “Defense acquisition 
revolves around 15-year programs, 5-year plans, 3-year management, 2-year congress, 
18-month technologies, 1-year budgets, and thousands of pages of regulations’’ (p. 1). 
Despite heavy oversight, the DoD continues to manage programs that are delivered late 
and that experience cost overruns, all while additional layers of oversight are inserted. 

2. History of Acquisition Failures 

The current management problems that acquisition programs face include the 
failure to control costs and schedules. According to defense acquisition specialist Moshe 
Schwartz (2010), “more recently, concerns over defense acquisitions have centered 
around significant cost overruns, schedule delays, and an inability to get troops in the 
field the equipment they need when they need if’ (p. 13), but acquisition and contracting 
failures are a persistent problem. During the Civil War, President Abraham Lincoln 
forced Secretary of War Simon Cameron to resign due to corruption and mismanagement 
of contracting in the War Department (The Lincoln Institute, 2011). The calamity of 
acquisition management has existed in the United States military since the Civil War, and 
as long as there are contracts, contracting oversight failures will be difficult to eliminate. 
Mitigating the cost and frequency of oversight failures through the sound use of contract- 
and program-level indicators of future variance may offer some relief from these failures. 
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3. Importance of Research and Development 

R&D activities are necessary to develop many technologies used in warfighting 
systems. R&D directly contributes to program success by demonstrating successively 
higher levels of integration and realism in a system’s or subsystem’s technology. This 
increasing level of demonstrated ability eliminates some technical risk. R&D for some 
systems and components begins before Milestone (MS) A. The R&D discussions in this 
research focus on later system R&D, specifically during the EMD acquisition phase. 

4. Current Acquisition Climate 

The proposed 2010 defense budget accounted for 20% of the federal budget, of 
which one third was for defense acquisitions (BENS, 2009). During July 2011, Congress 
debated cutting $866 billion from the defense budget across 10 FYs (Ewing, 2011). By 
comparison, the DoD requested a budget authority of $671 billion for FY2012 (USD 
Comptroller, 2011). The DoD spends approximately $400 billion a year on contracted 
goods and services (Carter, 2010). It appears the DoD must continue satisfying more 
requirements with similar budgets, or as Under Secretary Carter (2010) says, “do more 
without more’’ (p. 1). 

B. PURPOSE 

Our purpose in this research is to identify the presence of failure or success 
indicators before the production phase for DoD acquisition programs using both program 
and contract variances. To identify the indicators influential in program failure or 
success, we examine the variances during the production phase by analyzing statistical 
relationships of preproduction program and contract variances. We determined these 
relationships by examining the impact of cost, schedule, and engineering variances 
during research and development to later period variances of program execution. 

CV, SV, and EV within individual programs have only been qualitatively linked. 
Under Secretary Carter (2010) noted. 

As all programs compete for funding, the usual result is that a program 

settles into a level-of-effort pattern of annual funding that does not deviate 

much from year to year. The total program cost is the level-of-effort times 
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the total length of the program. Thus a one-year extension of a program 
set to complete in 10 years can be expected to result in a 10 percent 
growth in cost as the team working on the project is kept on another year. 

(p. 4) 

Further understanding the interrelationship of CV, SV, and EV could allow program 
managers to better understand the full programmatic impact of a cost, schedule, or 
technical problem during EMD. Identifying outliers that defy the norm of a positive CV, 
SV, and EV reinforcing loop could also aid future researchers in identifying best 
practices in recovering from a program cost overrun, schedule slip, or technical 
deficiency. A reinforcing loop exists when CV, SV, and EV interrelate and cause each 
other to continuously rise in a reinforcing, rather than stabilizing manner. The cost and 
schedule variances that are part of the earned value contract reports in SARs may also be 
linked to the program-level variances. 

Performing a quantitative analysis on program and contract variances could 
provide statistical predictions of future effects. The results will help at all levels of 
program management, including during the initial programming of requirements. The 
results will also show which contract and program variances have the greatest effect, 
thereby helping program managers to prioritize which variances to give their finite 
attention. Beyond this, formulating and estimating this linkage will enhance 
understanding of programs and contracts and will close gaps between the closely related 
acquisition communities of program management and contracting. 

C. RESEARCH QUESTIONS 

The problems with MDAPs running over budget and behind schedule gave rise to 
several research topics that must be investigated. Of particular interest to us is the 
increased use of fixed-price contracts recommended by DoD and the effects of contract 
variances on program variances. In order to focus these general research topics, the 
following questions were proposed. 

1. Primary Questions 

1. What effect does fixed-price R&D have on production cost, schedule, and 
technical performance? 
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2. Do different segments of MDAPs (e.g., fighters, tanks, missiles, satellites) 
exhibit differing cost and schedule growth? 

3. Does early CV, SV, or EV serve as a leading indicator of later period CV, 
SV, and/or EV in either EMD or post Milestone C? 

2. Secondary Questions 

1. What portion of MDAPs have fixed-price incentive R&D contracts? 

2. Is there qualitative information to support the assertion that fixed-price 
contracts during the EMD phase hinder the identification of program problems? 3.If the 
effect of fixed-price R&D is measurable, are the variances larger with regard to cost, 
schedule, or engineering during EMD and production? 

4. Based on the results found in this research, can any definitive policy 
recommendations be made? 

D. SCOPE AND METHODOLOGY 

In this research, we include the following: (1) a review of general acquisition CV, 
SV, and EV performance; (2) an in-depth review of the effect of fixed-price R&D during 
EMD on program CV, SV, and EV; and (3) a discussion of the interrelationships of CV, 
SV, and EV. We conclude the research with a recommendation for R&D contract type 
and a discussion of the implications of overall CV, SV, and EV performance. 

In this research, we first examine the available archival MDAP data using 
traditional descriptive statistics in order to determine whether identifiable patterns exist 
among contracts within MDAPs. This analysis helps close existing gaps between our 
understanding of program management and contract management. In order to perform 
this analysis, we collected cost, schedule, and engineering variance data, including post 
MS C production and deployment phase data, for 31 MDAPs from Selected Acquisition 
Reports (SARs) for multiple years. The database consists of cross-sectional, time-series 
data. The cross-section includes multiple programs and their attendant contracts during a 
particular year. The time-series consists of individual programs and their attendant 
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contracts identified by year over a several-year period. We use this dataset, when 
combined, for cross-sectional, time-series analysis, also known as panel data analysis 
(Princeton University, 2007). 

Following the descriptive statistics analysis, we employ the linear multiple 
regression analysis that combines cross-sectional, time-series SAR data by examining the 
effect of specified explanatory variables on an outcome measure. Additionally, we 
propose future research using structural equation modeling (SEM), a method that allows 
the simultaneous method of multiple factors. SEM analysis permits the inclusion of 
conceptually constructed variables that underlie the measurable variables, thereby, 
helping to explain the full effect of fixed-price incentive R&D contracts within MDAPs 
on later period cost and schedule variance during the production and deployment phase. 
The use of a structural equation model would allows others to examine variance in ways 
not previously available; that is, it would allow the consideration of multiple, interrelated, 
and simultaneously varying factors such as the cost, schedule, and engineering variances 
of a program and the earned value cost and schedule variances of that program’s 
contracts. 

As indicated, our objective in this research is to determine the likely effects of the 
increased usage of fixed-price R&D contracts during EMD on long-term program cost, 
schedule, and engineering variances. Although we give SARs particular emphasis, we 
also utilize DAES reports and Federal Procurement Data System (FPDS) data from the 
Defense Acquisition Management Information Retrieval System (DAMIRS). The 
methodology we used in this research consisted of the following steps: 

1. Conduct a literature review. 

2. Collect historical data from DAMIRS. 

3. Build a database that includes both program data and contract data by 
year, with all required fields in sufficient number for analysis using SPSS 
(a statistical package that permits cross-sectional, time-series analysis of 
measurement variables). 

4. Conduct a descriptive statistical analysis. 
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5. Complete an initial model for CV, SV, and EV. 

6. Complete an initial model pooling of CV, SV, EV, and other relevant 
program and cost and schedule contract variances measured in dollars. 

7. Conduct a thorough investigation of the program and contract variances 
using regression analysis to identify explanatory variables that influence 
these measures. 

8. Prepare a recommendation regarding R&D contract type selection. 

E. ORGANIZATION OF STUDY 

In Chapter II (Background), we provide an explanation of the current DoD 
acquisition process, the DoD acquisition categories, and different contract types. We 
provide a summary of the relationship between contract types and the acquisition phases 
and a description of the possible risk experienced during different phases of a program. 
In this chapter, we also include sections discussing the importance of R&D along with 
technology readiness levels. 

In Chapter III (Purpose), we provide explanations of the regulations that govern 
acquisition variances and aid in the decision-making of acquisition professionals, 
including the Nunn-McCurdy Amendment, Acquisition Program Baselines (APBs), 
SARs, and earned value management. We include a definition of each of the program 
and contract variances utilized in this research. We also provide in Chapter III the 
information required to understand program- and contract-level variances, including 
program cost, schedule, and engineering variances, as well as contract earned value cost 
and schedule variances. We conclude the chapter with an explanation of why the study 
of these acquisition variances is important. 

In Chapter IV (Database Construction and Analysis), we describe the process we 
used to collect data and to perform the analysis of variance. The chapter contains the 
steps we followed, including our data collection methods and the use of descriptive 
statistics. 
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In Chapter V (Statistical Methodology and Results), we provide an explanation of 
our methodology and analysis. The regression analysis methodology is discussed. This 
analysis, conducted using cross section, time series methods, should be viewed as 
exploratory, in nature, and certain aspects of our results need to be interpreted in this 
light. This includes our methodology for cross-sectional, time-series analysis. Our 
analysis includes identifying the relationship between preproduction program and 
contract variances and future program variances. The interrelationships we describe will 
help acquisition professionals better understand both the effects of cost, schedule, and 
engineering variances on programs and the linkage between programs and associated 
programs. 

In Chapter VI (Summary, Conclusions, and Recommendations), we explain the 
findings of our research and discuss the implications of the findings to practitioners. We 
then draw conclusions based on those results. We also provide recommendations for 
future research efforts and disclose the limitations of the study. 

F. BENEFITS OF THE STUDY 

We anticipate that this research will help acquisition personnel develop a better 
understanding of program and contract cost and schedule variances, and possibly provide 
indications of future program cost increases to establish better management techniques. 
Understanding the implications of fixed-price incentive R&D acquisition programs on 
cost, schedule, and engineering variances will help the DoD set appropriate policies for 
contract type in the early stages of system development. In this research, we provide the 
first comprehensive analysis of both the relationship between program data and contract 
data. One reason why understanding this relationship is so important is that problems 
with cost growth and schedule slippage are typically reported at the program level but 
contract types applies at the contract level, and a program may contain several different 
contracts during both the EMD phase and the production phase. This research will also 
help the DoD more appropriately consider the long-term impacts of proposed 
procurement cuts and will help leaders determine the best structure of those cuts. It may 
also improve system delivery time to the warfighter. 
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II. BACKGROUND 


A. OVERVIEW 

In this chapter, we provide the background information required to understand the 
scope of this research, including a review of the DoD acquisition process and three 
decision support systems. We briefly explain federal contract types applicable to 
acquisitions, along with the preferred contract types for each phase of the DoD 
acquisition process. Then, we explore the importance of R&D to acquisition program 
success, and we discuss technology readiness levels. The risk associated with contract 
type and acquisition phase is reviewed. We conclude the chapter with a discussion of the 
primary documents from which the study draws data. 

B. DEFENSE ACQUISITION PROCESS 

In this section, we provide an overview of the DoD acquisition process and the 
three principal decision support systems: the Planning, Programming, Budgeting, and 
Execution (PPBE) process; the Joint Capabilities Integration and Development System 
(JCIDS); and the Defense Acquisition System. These three integrated systems ensure 
acquisition personnel make consistent decisions to efficiently acquire the right products 
and services required by the warfighter. The DoD’s decision support systems, described 
in the Defense Acquisition Guidebook (DoD, 2011), and its policies, oversight, and 
integration can be seen in Figure 1. The boxes in Figure 1 annotate the direct supporting 
references for each system. The overlap in the three systems can clearly be seen in the 
figure and is important to understanding the balance of all three systems in the defense 
acquisition process. 
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Figure 1. The DoD Decision Support Systems 
(DoD, 2011) 

1. The Planning, Programming, Budget, and Execution (PPBE) Process 

The PPBE process provides for strategic planning, program development, and 
resource determination. The PPBE process generates funding appropriations to support 
DoD acquisition requirements. During this process, the Secretary of Defense provides 
priorities, which guide resource allocation decisions. Program managers must be aware 
of this process to effectively manage the funding of contracts. Failure to recognize the 
link between budgets and requirements in the acquisition process may directly result in 
poor performance during the acquisition process (Gansler, 2002). 

Accurate program estimates are important in order to obtain necessary funding 
during the budgeting process. The timing of funding availability is also important and 
could significantly affect program schedules and costs. In an address to the Senate 
Armed Services Committee regarding the FY2011 Continuing Resolution, Secretary of 
Defense Robert Gates stated that the failure of timely funding would damage 
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procurement and research programs, causing delays, increasing costs, and disrupting 
production of highly demanded assets (Gates, 2011). 

2. Joint Capabilities Integration and Development System (JCIDS) 
Process 

The PPBE process receives development and production life cycle estimates from 
the requirements defined in the Joint Capabilities Integration and Development System 
(JCIDS) process. JCIDS procedures support the Chairman of the Joint Chiefs of Staff 
and the Joint Requirements Oversight Council (JROC) in determining joint military 
capability needs. JCIDS is the DoD’s systematic means for appraising gaps in military 
warfighting capabilities and for proposing solutions to solve those gaps. Announced in 
2003, the JCIDS process changed the way requirements were identified—replacing 
Service-centric requirements processes with a joint capabilities system. The change came 
after Secretary of Defense Donald Rumsfeld sent a memo that stated: 

Please think through what we all need to do ... to get the requirements 
system fixed. It is pretty clear it is broken, and it is so powerful and 
inexorable that it invariably continues to require things that ought not to be 
required, and does not require things that need to be required. (Chairman 
of the Joint Chiefs of Staff [CJCS], 2006, p. 5) 

The elimination of the word requirements from the generation system signaled the 
DoD’s intent of the JCIDS process to determine possible procedural or training-based 
solutions (non-material solutions) available along with material solutions and to justify 
the need for the capability (CJCS, 2006). 

The JCIDS process is covered in CJCS Instruction 3170.01G (CJCS, 2009) and is 
summarized here to provide for an understanding of the requirements review process in 
defense acquisitions. The process implemented in JCIDS supports the management of 
resource investments and must be performed prior to the acquisition process for new 
systems to begin. Figure 2 illustrates the acquisition process and what documents are 
produced during the JCIDS process in order to identify gaps in capabilities (CJCS, 2009). 
As seen in Figure 2, various parties are involved in the process and are required to 
produce multiple coordinating documents. 
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□ = Sponsor Activity JCIDS Document Acquisition decision 

Figure 2. JCIDS and Defense Acquisition 
(CJCS, 2009) 

The JCIDS process incorporates the military missions and the capabilities 
required to perform the operational objectives associated with these missions. The 
incorporation of capabilities into the JCIDS process originates through a capabilities- 
based assessment (CBA) and an identified valid gap in mission requirement. As stated by 
General McChrystal, 

if only non-materiel solutions are recommended or a non-materiel solution 
can be implemented independent of proposed materiel needs, a joint 
doctrine, organization, training, materiel, leadership and education, 
personnel, or facilities (DOTMLPF) change recommendation (DCR) is 
produced. (CJCS, 2009) 

An initial capabilities document (ICD) identifies gaps requiring materiel solutions. An 
approved ICD provides the required information to form the materiel development 
decision (MDD) and starts the actual acquisition process (CJCS, 2009; see Figure 2) 

3. Defense Acquisition System Process 

The objective of the defense acquisition system is to acquire products that satisfy 
specified needs and provide measurable improvements to mission capabilities at a fair 
and reasonable price (DoD, 2011). Defense acquisition follows distinct program phases, 
a series of decision points, and significant milestones. The process defined in the DoD 
5000 Series that guides acquisition programs for MDAPs is summarized here. Figure 3 
illustrates the defense acquisition system process by showing the relationship between 
each phase and milestone. 
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Figure 3. The Defense Acquisition Management System 
(USD[AT&L], 2008) 

a. Material Solution Analysis Phase and Milestone A 

After an approved ICD and a MDD, the milestone decision authority 
(MDA) may authorize entry into the acquisition management system, starting with the 
material solution analysis phase, also known as the concept refinement phase (see Figure 
3). For MDAPs, a defense acquisition board (DAB) is formed that provides advice on 
critical acquisition decisions. The DAB is chaired by the USD(AT&L) and includes 
senior officials from the Joint Staff, the military departments, and the staff offices within 
the OSD. The USD(AT&L) is the MDA when there is a DAB. The purpose of the 
material solution analysis phase is to perform an analysis of alternatives (AoA) for all 
potential material solutions; the phase culminates with a decision about whether to 
proceed with MS A as designated by the MDA. An AoA assesses each alternative on 
measures of effectiveness, cost, schedule, concept of operations, and overall risk. The 
material solution analysis phase ends once the MDA approves the results of the 
completed AoA (USD[AT&L], 2008). 
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b. Technology Development Phase and Milestone B 

The acquisition decision memorandum is signed at MS A, documenting an 
approved material solution, a technology development strategy (TDS), and entry into the 
technology development phase. The TDS establishes the preliminary acquisition plan, 
including the cost, schedule, and performance goals for the engineering and 
manufacturing development phase. The purpose of the technology development phase is 
to reduce technology risk and to establish subsystems and components that must be 
demonstrated before being fully integrated into a system (USD[AT&L], 2008). 

This phase is a continuous discovery and development process to identify 
the accessibility and refinement of the requirement. Competitive prototypes are built 
based on initial capabilities. The users, or the relevant Service, should establish a 
capabilities development document (CDD) during the technology development phase that 
specifies the technical performance parameters required to deliver the proposed design 
and that fills the capabilities gap identified in the ICD. 

The CDD replaced the operational requirements document (ORD) that 
was used under the old requirements system. The CDD supports the MS B decision and 
must be approved prior to MS B. It guides the engineering and manufacturing 
development phase by defining measurable and testable capabilities. The CDD identifies 
operational performance attributes of the proposed system known as key performance 
parameters (KPPs). KPPs make up the bulk of the CDD and list each required measure 
of effectiveness. The parameters contain both threshold (required or minimum) and 
objective (desired or maximum) performance values (USD[AT&L], 2008). 

The program manager should prepare for a preliminary design review 
(PDR) prior to MS B, as planned by the development strategy. The system developers, 
engineers, users, and certification authorities should all collaborate to agree on a 
proposed solution based on demonstrated technology that meets the PDR. The PDR 
report and the demonstrated manufacturing process are provided to the MDA at MS B to 
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identify projected cost, schedule, and risk. At MS B, the MDA decides to proceed to the 
next acquisition phase, perform additional work, or terminate the effort (USD[AT&L], 
2008). 


c. Engineering and Manufacturing Development Phase and 
Milestone C 

The engineering and manufacturing development (EMD) phase begins 
with an approved acquisition strategy at MS B. MS B and entrance into the EMD phase 
typically mark the initiation of an acquisition program. The purpose of the EMD phase is 
to develop an integrated system of demonstrated subsystems and components design. In 
this phase, a proven system capability is also developed, and an achievable and 
affordable manufacturing process is demonstrated (USD[AT&L], 2008). 

The integrated system design defines functionality and interfaces; provides 
a complete, detailed design; and reduces full system-level risk. The system capability 
and manufacturing process demonstration assures that the system will operate in 
accordance with performance parameters and demonstrates that system production can be 
supported. This phase ends when the system meets all approved requirements, performs 
in its intended environment, effectively demonstrates manufacturing capabilities, has 
reasonably available production capabilities, meets MS C requirements, and has the 
approval of the MDA to commit the program or to terminate the effort (USD[AT&L], 
2008). 


d. Production and Deployment Phase 

The production and deployment phase is the final phase in the systems 
acquisition process (see Figure 3). The purpose of this phase, according to DoD 
Instruction 5000.02, is “to achieve an operational capability that satisfies mission needs” 
(USD[AT&L], 2008, p. 26). MS C approval begins the production and deployment 
phase. MS C specifically authorizes low-rate initial production (LRIP), which is 
performed at the beginning of the phase. In the past, the production and deployment 
phase did not begin until the start of full-scale production, at MS III. For the purposes of 
this research, we considered the beginning of the production and deployment phase as 
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occurring at LRIP approval, even for programs that did not have a milestone review at 
that point. We provide further information regarding our data collection methods and 
analysis in later chapters. During LRIP, manufacturing capability is verified. Initial 
operational testing and evaluation is performed prior to the point that the MDA 
authorizes entry into full-rate production and deployment from a successful full-rate 
production decision review (USD[AT&L], 2008). 

4. Acquisition Program Categories 

Acquisition programs are assigned to an acquisition category (ACAT) based on 
their location in the acquisition process, on cost, or on whether the program is of special 
interest to the MDA. The USD(AT&L) can reclassify an acquisition program at any 
time. The discussion in the following subsections explains the differences between each 
category. 


a. ACAT I Programs 

ACAT I programs, also called Major Defense Acquisition Programs 
(MDAPs), include programs designated by the MDA as special interest. According to 10 
U.S.C. § 2430 (2011), MDAPs are designated by the Secretary of Defense and estimated 
to require a total expenditure for RDT&E of more than $300 million in FY1990 dollars or 
for procurement of more than $1.8 billion in FY1990 dollars. ACAT I programs are 
further divided by decision authority. The programs with the highest interest are ACAT 
ID and the MDA is the USD(AT&L). For ACAT IC, the MDA is head of the DoD 
Component or, if delegated, the Component Acquisition Executive (CAE; USD[AT&L], 
2008). 


b. ACAT I A Programs 

ACAT IA programs are also called Major Automated Information 
Systems (MAIS) and, in some cases, meet the definition of an MDAP, but also include 
programs designated by the MDA as special interest. According to 10 U.S.C. § 2445 
(2011), MAIS programs are designated by the Secretary of Defense when program 
expenditures in FY2000 constant dollars are estimated to exceed the following amounts: 
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$32 million for all expenditures directly related to automated information systems 
definition, design, development, and deployment costs in a single FY; $126 million for 
all expenditures directly related to the entire program; or $378 million for all 
expenditures related to the total life-cycle costs. AC AT IA programs are further divided 
into ACAT lAM, in which the MDA is the USD(AT&L) or his or her designee, and 
AC AT I AC, in which the MDA is the head of the DoD Component or, if delegated, the 
CAE (USD[AT&L], 2008). In this research, we examine only MAISs that are also 
MDAPs. 


c. ACAT II and ACAT HI Programs 

ACAT II programs do not meet criteria for ACAT I, but they are still 
major systems with a dollar value in FY2000 constant dollars estimated by the DoD 
Component Head of eventual RDT&E greater than $140 million or of procurement of 
more than $660 million. The MDA for ACAT II programs is the CAE or an individual 
designated by the CAE. ACAT III programs include DoD programs that do not meet 
ACAT II criteria, and the MDA is designated by the CAE (USD[AT&L], 2008). We do 
not directly examine ACAT II and ACAT III programs in this research, although it may 
be possible to generalize some of the findings of this research to these programs when 
sufficient similarity in treatment exists. 

C. FEDERAL CONTRACT TYPES 

A wide variety of contracts can be used to purchase products and services 
required by the federal government. A large acquisition program typically has multiple 
types of contracts, and selecting the right contract type for an acquisition is essential to 
successful completion. Federal Acquisition Regulation (FAR, 2011) part 16 describes 
the different federal contract types. Contracts vary according to the responsibility 
assumed by the contractor for the costs of performance and the amount and nature of 
profit incentives offered to the contractor for specified standards or goals (FAR, 2011). 
For DoD MDAPs and this research, the two general categories of contracts are fixed- 
price and cost-reimbursement, but there are other types outside these two categories. 
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1 . 


FIXED-PRICE CONTRACTS 


Fixed-price contracts are suitable when acquiring supplies and services that users 
can describe in sufficient detail. The price is agreed upon during the award phase, or, in 
appropriate circumstances, an adjustable price may be included. Fixed-price contracts 
providing an adjustable price may include a ceiling price, a target price, or a combination 
of both. Under a fixed-price contract, most of the performance and cost risks are placed 
on the contractor though the use of incentives to control costs (FAR, 2011). The 
following summaries of the different types of fixed-price contracts are based on FAR 
(2011) part 16. 


a. Firm-Fixed-Price Contracts 

Firm-fixed-price (FFP) contracts are used when a fair and reasonable price 
can be determined at the beginning of the contract (FAR, 2011). The government pays 
the negotiated amount regardless of the contractor’s actual performance costs. 
Administrative requirements are not eliminated for FFP, but the burden is reduced. 
Given well-defined specifications, the contractor bears most of the risk. This contract 
type is preferred above all others because it encourages the contractor to contain costs 
(Garrett, 2007). The government prefers FFP if clear objectives exist and accurate 
pricing data is available, but that may not always be the case. 

b. Fixed-Price Incentive Contracts 

Fixed-price incentive (FPI) contracts are used when the government wants 

to incentivize technical performance and cost controls. Parties can negotiate a target cost, 

target profit, and a ceiling price that provides for the contractor to assume an appropriate 

share of the risk. If the contractor reaches the ceiling price as a result of an overrun, the 

contract essentially becomes an FFP contract. In principle, the contractor is paid no more 

than the ceiling price and must meet the requirements of the contract. The profit is 

adjusted by calculating the final price using a formula based on the relationship of final 

negotiated and target costs. Two different types of fixed-price incentive contracts are 

authorized: fixed-price incentive firm (FPIF) target contracts and fixed-price incentive 

successive (FPIS) targets contracts (FAR, 2011). Currently, there is a proposed Defense 
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Federal Acquisition Regulation Supplement (DFARS, 2011) rule to encourage the 
increased use of fixed-price incentive contracts. The origination of this rule is credited to 
a memo sent by the USD(AT&L) on November 3, 2010, that gave direction on attaining 
better efficiency and productivity in defense contract spending (Levin, 2011). In this 
study, we analyze this type of contract closely and identify the size of variances occurring 
compared to other types of contracts. 

2. COST-REIMBURSEMENT CONTRACTS 

Cost-reimbursement contracts allow the government to pay the contractor all 
allowable incurred costs that are fair and reasonable as prescribed in the contract. This 
type of contract is used when many uncertainties associated with contract performance 
and costs cannot be estimated with sufficient accuracy to use a fixed-price contract. A 
cost-reimbursement contract provides an estimate of the total costs for the purpose of 
obligating funds, and it establishes funding ceilings that cannot be exceeded without 
approval from the contracting officer. A contractor may exceed these ceilings at their 
own risk. Cost-reimbursement contracts are typically renegotiated or terminated if total 
costs exceed ceilings. 

Cost-type contracts place most of the cost and performance risk on the 
government (FAR, 2011). The contractor should put forth its best effort, but there is no 
promise of results. The government may not end up with the final product or service it 
contracted for, but it must reimburse the contractor for costs incurred that do not exceed 
the approved funding ceilings. The government is obligated to reimburse all actual costs 
that are allowable, allocable, and reasonably incurred to the extent prescribed in the 
contract (FAR, 2011). In the following subsections we summarize the different types of 
cost contracts; these summaries are based on FAR (2011) part 16. 

a. Cost-Plus-Fixed-Fee Contracts 

Cost-plus-fixed-fee (CPFF) contracts pay a pre-determined, fixed-fee that 
is agreed upon during contract negotiations. The fee may be adjusted for changes in the 
work to be performed (FAR, 2011). The contracting officer is responsible for monitoring 
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the contractor’s expenditures and can request an audit of the contractor’s vouchers at any 
time, similar to other cost-reimbursement auditing. 

b. Cost-Plus-Incentive-Fee Contracts 

Cost-plus-incentive-fee (CPIF) contracts are used to encourage contractors 
by providing greater profits through cost savings and other performance improvements. 
The government pays allowable, reasonable, and allocable costs and an incentive fee 
based on the contractor’s achievement of the objectives calculated by using a formula 
that is based on the relationship of total allowable and target costs. The contract specifies 
target costs and target fees and establishes minimum and maximum fees and a fee- 
adjustment formula. The increase or decrease in fee is intended to incentivize the 
contractor to effectively manage costs (FAR, 2011). 

c. Cost-Plus-Award-Fee Contracts 

Cost-plus-award-fee (CPAF) contracts are used to provide additional 
incentives to contractors to achieve excellence in areas such as quality, timeliness, 
technical ingenuity, and cost-effective management (FAR, 2011). The government pays 
allowable costs, a base fee, and an award fee based on a subjective evaluation of 
performance. The contract provides for interim rating periods during contract 
performance. It is possible that the inclusion of KPPs in an MDAP’s CDD at MS B may 
induce the program manager to incentivize KPPs through the use of award fee incentives 
(Hildebrandt, 2010) since award fees offer the ability to incentivize ambiguously defined 
objectives. 

D. TYPICAL CONTRACT TYPE BY ACQUISITION PHASE 

The government can use a wide variety of contracts to order required products 
and services. Selection of the contract type is driven by risk considerations (FAR, 2011), 
which vary among programs and across acquisition life cycle phases (Garrett, 2007). 
Certain contract types are more suited to certain types of acquisitions and certain stages 
of the acquisition life cycle (FAR, 2011; Defense Acquisition University [DAU], 2008). 
No particular contract type is the answer for any phase, but enduring trends in selection 
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of contract type exist due to the limitations typically present during each life-cycle stage. 
Preferred contract types for a phase may shift over time due to changing policy or 
guidance or to shifting praetitioner preference based on research regarding programs of a 
similar nature. As we explain in more detail later in this section, contract type is 
ultimately a collaborative decision between a program’s contracting officer and program 
manager, subject to negotiation with a program’s contractors (FAR, 2011; DFARS, 2011, 
part 234). 

1. RISK AND PREFERRED CONTRACT TYPE 

Fixed-price contracts are typically preferred by the government to minimize its 
risk, but they may not be appropriate if the work lacks precise specifieations or cost 
estimates. FAR subpart 16.101 eharaeterizes eontract types as those ranging “from firm- 
fixed-price, in which the contractor has full responsibility for the performance costs and 
resulting profit (or loss), to cost-plus-fixed-fee, in which the contractor has minimal 
responsibility for the performance eosts and the negotiated fee (profit) is fixed” (FAR, 
2011). If funding ends before eompletion, the contract should be renegotiated or 
terminated. Incentive contracts fall in between these two extremes. Fixed-price incentive 
and cost-plus-incentive-fee contracts should only be considered if it is in the 
government’s best interest to use cost and, when appropriate, performance ineentives 
(FAR, 2011). 

Cost estimates are ultimately predietions about the future, and they are subjeet to 
error (Garrett, 2007). Buyers and sellers understand that error is present in cost 
estimation. For the purposes of this research, we consider the government the buyer and 
the eontraetors the sellers. The government is wary of aecepting an exeessive eost for 
goods or services, and contractors seek to ensure that they do not suffer a loss on a sale. 
Each party attempts to hedge against its own risk. Contractors estimate pessimistically, 
and the government estimates optimistically, resulting in a “range of possible costs” 
(Garrett, 2007, p. 106). 

Whenever praetieable, the government’s default option is to use a firm-fixed-price 
contract, but this is frequently not possible. If a large amount of estimating uncertainty is 
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present, the government and a contractor may be unable to reach an agreement without 
the government accepting an excessively pessimistic cost estimate. This results in the 
government negotiating with a contractor to establish the contract’s type and price. This 
negotiation often results in the government sharing cost risk with the contractor to bring 
down the contracted price, although achieving a more reasonable price is not the sole 
objective. The objective of this negotiation “is to negotiate a contract type and price (or 
estimated cost and fee) that will result in reasonable contractor risk and provide the 
contractor with the greatest incentive for efficient and economical performance” (FAR, 
2011, subpart 16.103). Selection of contract type can be depicted as a trade-off between 
the buyer’s risk (risk to the government) and the seller’s risk (risk to the contractor), as 
shown in Figure 4. 


Types of Contracts 


Fixed-price 


Cost-reimbursement 


^^^^^^^—h 

FFP FP/EPA FPI CS CR CPIF CPAF CPFF 




Low ◄- Buyer’s risk -► High 

High ◄- Seller’s risk -► Low 

Figure 4. Types of Contracts by Risk 

This figure from Garrett (2007, p. 127) was altered to fit the context of this 
research, federal MDAPs. Accordingly, Time and Materials was eliminated from the 
graphic; although it is a permissible federal contract type, it is not significant with respect 
to MDAPs. Cost Plus Percentage of Cost was also eliminated because, by law, the Cost 
Plus Percentage of Cost contract type is no longer permissible for federal contracts. 

2. RISK BY ACQUISITION PHASE 

During the beginning stages of system acquisition, risk is typically higher. 
System acquisition risk is initially driven by a lack of program definition and by 
uncertainty regarding the program’s ability to meet scheduled technical achievements. 
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CDD is an attempt to change this. From a contracting perspective, risk can be understood 
as poorly or undefined requirements driving uncertainty in performance cost. When 
looking beyond a contracting risk perspective, acquisition risk can be understood as a 
combination of four factors: technical risk, cost risk, schedule risk, and programmatic 
risk (Blanchard & Fabrycky, 2006). 

Technical risk is the possibility that a technical requirement may not be met 
during a system’s life cycle (International Council on Systems Engineering [INCOSE], 
2004). For a government acquisition, this could be understood as a program’s failure to 
achieve one or more of the threshold requirements. Cost risk is the possibility that the 
specific budget allocated to a program will be exceeded (Blanchard & Fabrycky, 2006; 
INCOSE, 2004). Schedule risk is the possibility that a program will fail to meet 
scheduled milestones, and programmatic risk is the external risk posed by the program’s 
environment (INCOSE, 2004). Common causes for program cancellation are cost 
overruns, schedule slips, and failure to eliminate technological risk. Figure 5 depicts 
relationships between these risks, as proposed by the INCOSE. 



Eigure 5. Typical Relationships Among the Risk Categories 
(INCOSE, 2004, p. 65) 
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Technical risk is typically highest at the beginning of a program, when a 
program’s technical problems are least understood (INCOSE, 2004). Cost risk also 
typically decreases as requirements become more defined and as a program gains 
institutional support. This increase in support protects maturing programs from budget 
reductions. Schedule risk also normally decreases as a program develops, although the 
effect of a schedule slip increases as more resources are devoted to a program. FAR 
(2011) subpart 16.104(d) recognizes that acquisition risk is highest at the beginning of a 
program: 


Complex requirements, particularly those unique to the Government, 
usually result in greater risk assumption by the Government. This is 
especially true for complex research and development contracts, when 
performance uncertainties or the likelihood of changes makes it difficult to 
estimate performance costs in advance. 

3. ACQUISITION PHASE AND PREFERRED CONTRACT TYPE 


As a program progresses through the acquisition life-cycle process, risk typically 
decreases. During the concept refinement and technical development phases, risk is high. 
Critical technologies that are projected to mature may not (INCOSE, 2004), and newer 
programs may be more susceptible to “paying the bills” of another program (Software 
Engineering Institute [SEI], 2009). Approval to proceed from one acquisition phase to 
another comes from the MDA, as we explain in Chapter III. Figure 6 illustrates the 
typical contract type utilized in the acquisition process by life-cycle phase. 
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Figure 6. Contract Type by Life-Cycle Phase 
(DAU, 2008) 
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Developmental work typically requires cost-reimbursement contracts (FAR, 
2011). During the concept refinement and technology development phases, the 
predominant contract type is CPFF (DAU, 2008). This is in keeping with the higher risk 
associated with contract performance during those phases, although CPFF is not the only 
contract type used before MS B. R&D continues during the EMD phase, but by this 
point programs typically demonstrate enough technical maturity and design stability to 
permit contract types with less government cost risk. CPFF, CPAF, CPIF, FFP contracts 
are common during the EMD phase (DAU, 2008), along with FPIF contracts, which we 
examine in Chapter V. The EAR (2011) recommends that when follow-on production 
requirements have been contemplated for an R&D contract, contracts should progress 
from cost-reimbursement to fixed-price. System acquisition contracts during the 
production and deployment phase tend to be FPIF or FPAF, and sustainment contracts are 
often FPIF, FPAF, or FFP. 

Preferred contract types shift based on changing policy guidance, phase, or 
research. A study in Israel on 110 defense development projects found that cost-plus 
incentive fee contracts resulted in better performance when technological uncertainty was 
high at the start of the project (Sadeh, Dvir, & Shenhar, 2000). The study recommended 
a combination of cost-plus and fixed-price contracts for projects, moving toward fixed- 
price when uncertainty is reduced (Sadeh et al., 2000). Debate continues as to when 
uncertainty is reasonably low enough to regularly permit fixed-price contracts. A current 
DoD proposed rule would increase the use of FPIF contracts during EMD for the purpose 
of attaining better contract performance (Levin, 2011). In later chapters of this research, 
we examine the relative performance of contracts during production years on the basis of 
predominant contract type during EMD, with the intent of solving the debate regarding 
the merit of various contract types during EMD. 

Because the contracting officer is responsible for safeguarding the interests of the 
government (EAR, 2011, subpart 1.602-2), the ultimate responsibility to appropriately 
select a program’s contract type rests with him or her. The FAR (2011) states that, 
“selecting the contract type is generally a matter for negotiation and requires the exercise 
of sound judgment” (subpart 16.103(a)). With the exception of stating that the 
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contracting officer is responsible for determining the contract type through negotiation 
with contractors, the FAR is largely silent on who else should be involved in the decision, 
with the exception of R&D contracts. In practice, determining a program’s contract type 
is often a collaborative decision between a program’s contracting officer and program 
manager or contract user representative. This collaborative approach is recommended to 
contracting officers for R&D acquisitions in the FAR (2011) due to “the importance of 
technical considerations in R&D’’ (subpart 35.006(b)). DFARS (2011) subpart 
234.004(2) goes further and dictates that an MDAP’s MDA should select the 
development program contract type, taking into consideration the contracting officer’s 
recommendation, at the time of MS B approval. This DFARS exception to the typical 
practice of the contracting officer selecting the contract type reinforces the collaborative 
nature of determining contract type. 

The FAR (2011) provides “a wide selection of contract types ... in order to 
provide needed flexibility in acquiring the large variety and volume of supplies and 
services required by [the government]’’ (subpart 16.101). The flexibility given to 
contracting officers in determining contract types and the collaboration suggested in 
making the determination of contract type indicates the complex nature of selecting a 
contract type, regardless of the acquisition phase. Accordingly, no particular contract 
type is the universal answer for any acquisition phase. 

E. IMPORTANCE OE RESEARCH AND DEVELOPMENT 

Four main causes of program failure have been identified in other research, 
including requirements changes, budget instability, technology risk, and poor execution 
due to inadequate program decision-making information (Miller, 2008). R&D reduces 
technology risk for system acquisition programs and is thus critical to program success 
both during and after the main acquisition phases associated with R&D, which are as 
follows: concept refinement, technology development, and system development and 
demonstration. Two areas that contribute to the relative success of programs during and 
after the R&D acquisition phases are R&D contracting and technology readiness. 
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1 . 


R&D CONTRACTING 


Program managers must use careful consideration when determining the type of 
contract used for R&D efforts. R&D contracts should encourage high creativity and 
innovation. The FAR (2011) states that “the primary purpose of contracted R&D 
programs is to advance scientific and technical knowledge and apply that knowledge to 
the extent necessary in order to achieve agency and national goals” (subpart 35.002). 
Contracting officers must judiciously apply the FAR policies on contract type to achieve 
the government’s purpose of R&D efforts. MDAPs are only responsible for a portion of 
R&D spending within the DoD. Much of this non-MDAP spending is devoted to basic 
research, including technology exploration that takes place long before technologies 
mature to the point necessary to begin MDAPs. The primary purpose of MDAP R&D 
contracting is to improve the technology readiness of a system’s contemplated critical 
technologies through developmental and applied research. 

One of the imperatives of R&D contracting is to establish a full understanding 
between the parties regarding the intent of the R&D effort (FAR, 2011). This 
understanding can be impaired due to misunderstanding on the part of either party to the 
contract, resulting in R&D efforts that do not meet the needs of the government. Due to 
the uncertainty and ambiguity present in R&D efforts, the FAR (2011) instructs 
contracting officers to avoid sealed bidding. Contracting officers should instead utilize 
competitive negotiations with offerors to establish a comprehensive mutual 
understanding of the project (FAR, 2011). 

Providing flexibility with minimal administrative burden should be considered by 
contracting officers (FAR, 2011), but clear objectives must be conveyed for program 
success. Contracting officers should utilize technical personnel to define clear objectives 
for R&D projects (FAR, 2011). The DoD Space-Based Infrared System (SBIRS) is an 
example of negative results due to unclear objectives. The GAO reported that the cost 
overruns and schedule delays for SBIRS began at the inception of the program due to 
immature technologies and unclear requirements (GAO, 2003). In keeping with the FAR 
(2011), it is critical to successful developmental and applied research to encourage 

contractors to “exercise innovation and creativity” in responding to “clear and complete 
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... end objectives” (subpart 35.005). A well-defined research objective, combined with a 
clear understanding between the government and contractor, contributes to the relative 
success of a program by removing a potential impediment to MDAP R&D success. 

2. TECHNOLOGY READINESS LEVELS 

The level of uncertainty during R&D varies greatly, particularly during the early 
phases of a program. Improved technology readiness is theorized to contribute to the 
success of MDAPs during and after the R&D acquisition phases. Some researchers 
suggest that the relative maturity of an MDAP’s technologies contributes to improved 
cost and schedule variance outcomes. 

Some programs begin R&D with more mature technologies, but others have 
immature technologies. Program technologies must advance to certain minimum levels 
before programs are permitted to move beyond the technology development and EMD 
phases (DoD, 2011). Making consistent, informed judgments regarding the maturity of 
an MDAP’s technology requires a uniform system of measurement. One method of 
defining the maturity of a program’s technology is using technology readiness levels 
(TRLs) to rate the readiness of a technology against set criteria. Table 1 includes a 
summary of the TRL descriptions as shown in section 10.5.2 of the Defense Acquisition 
Guidebook (DoD, 2011). 
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Table 1. TRL Descriptions 
_(DoD, 2011)_ 


Technology Readiness Level 

Description 

1. Basic principles observed 
and reported. 

Lowest level of technology readiness. Scientific research begins to be 
translated into applied research and development. Examples might 
include paper studies of a technology's basic properties. 

2. Technology concept and/or 
application formulated. 

Invention begins. Once basic principles are observed, practical 
applications can be invented. Applications are speculative and there may 
be no proof or detailed analysis to support the assumptions. Examples 
are limited to analytic studies. 

3. Analytical and 
experimental critical function 
and/or characteristic proof of 
concept. 

Active research and development is initiated. This includes analytical 
studies and laboratory studies to physically validate analytical 
predictions of separate elements of the technology. Examples include 
components that are not yet integrated or representative. 

4. Component and/or 
breadboard validation in 
laboratory environment. 

Basic technological components are integrated to establish that they will 
work together. This is relatively "low fidelity" compared to the eventual 
system. Examples include integration of "ad hoc" hardware in the 
laboratory. 

5. Component and/or 
breadboard validation in 
relevant environment. 

Eidelity of breadboard technology increases significantly. The basic 
technological components are integrated with reasonably realistic 
supporting elements so it can be tested in a simulated environment. 
Examples include "high fidelity" laboratory integration of components. 

6. System/subsystem model or 
prototype demonstration in a 
relevant environment. 

Representative model or prototype system, which is well beyond that of 
TRL 5, is tested in a relevant environment. Represents a major step up in 
a technology's demonstrated readiness. Examples include testing a 
prototype in a high-fidelity laboratory environment or in simulated 
operational environment. 

7. System prototype 
demonstration in an 
operational environment. 

Prototype near, or at, planned operational system. Represents a major 
step up from TRL 6, requiring demonstration of an actual system 
prototype in an operational environment such as an aircraft, vehicle, or 
space. Examples include testing the prototype in a test bed aircraft. 

8. Actual system completed 
and qualified through test and 
demonstration. 

Technology has been proven to work in its final form and under expected 
conditions. In almost all cases, this TRL represents the end of true 
system development. Examples include developmental test and 
evaluation of the system in its intended weapon system to determine if it 
meets design specifications. 

9. Actual system proven 
through successful mission 
operations. 

Actual application of the technology in its final form and under mission 
conditions, such as those encountered in operational test and evaluation. 
Examples include using the system under operational mission conditions. 
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The use of TRLs is credited with enabling “consistent, uniform, discussions of 
technical maturity across different types of technologies” (DoD, 2011, p. 853). TRLs are 
limited, however, to maturity and do not directly address the probability that a part or 
subsystem will achieve the maturity required for system production. TRLs are also 
assessed on an ordinal scale, so it is incorrect to assume that a system will require 
equivalent effort to increase from one level to the next. Current DoD policy requires 
MDAPs to achieve TRL 6 by MS B and TRL 7 by MS C (DoD, 2011). The Defense 
Acquisition Guidebook also instructs MDAs to “consider the recommended TRLs ... 
when assessing program risk” (DoD, 2011). However, TRLs should not be the sole 
measure of a program’s technical risk because they do not measure the potential impact 
of failing to achieve technology maturity. TRLs cannot offer a full assessment of 
program risk, but they do provide decision-makers with uniform, comprehensible data 
regarding a program’s present technology maturity. 

Miller (2008) blamed requirements changes, technology risk, and poor execution 
due to insufficient or improper program decision-making information as three of the 
causes of poor performance for MDAPs. Programs entering the technology development 
phase with higher minimum and average TRLs have been shown to exhibit lower cost 
growth and schedule growth (Dillard & Ford, 2009). Higher levels of technology 
readiness could operate in an inverse relationship to technology risk: an MDAP with 
increased TRLs relative to its acquisition phase would then exhibit lower technology risk. 
Increasing the impartial use of TRLs could also improve program decision-making 
(GAO, 1999). It is therefore possible that higher TRLs relative to an MDAP’s 
acquisition phase could possibly mitigate Miller’s (2008) causes of poor performance. 

Improved technology readiness contributes to the success of programs during and 
after the R&D acquisition phase. Recent evolutionary acquisition efforts within the DoD 
have focused on using more mature technologies to develop incremental capabilities. 
Some researchers suggest that programs with higher TRLs tend to exhibit better cost and 
schedule performance (Dillard & Ford, 2009). Differences in TRLs may explain some of 
the variation not otherwise explained by the variables we model in later chapters. 
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F. SELECTED ACQUISTION REPORTS 

Selected Acquisition Reports (SARs), which are published annually, are one of 
the main sources of data we used for the analysis in this study. These reports provide a 
snapshot in time for each program we analyzed. As mandated by Congress under 10 
U.S.C. §2432 (2011), the Secretary of Defense must submit SARs for all MDAPs or 
programs designated as high interest by the USD(AT&L). Congress utilizes these reports 
to track the progress of MDAPs, specifically to detect early warnings of cost or schedule 
overruns (DoD, 2011). 

SARs summarize the latest status of total program cost, schedule, and 
performance as well as program unit cost and unit cost breach information. Each SAR 
provides a full life-cycle cost analysis for the reporting program and is prepared annually 
in conjunction with submission of the President’s budget. Subsequent quarterly 
exception reports are required only for those programs experiencing unit cost increases of 
15% or more, schedule delays of six months or greater since the current estimate reported 
in the previous SAR, or when MS B or MS C approval occurs within the reportable 
quarter (DoD, 2011). 

G. DEFENSE ACQUISITION EXECUTIVE SUMMARIES 

Defense Acquisition Executive Summary (DAES) reports are published quarterly 
and are the other source of data for the analysis we conducted in this research. DAES 
reports provide a comprehensive summary of AC AT I and AC AT lA programs between 
milestone decision points. The DAES reports must contain program assessments, unit 
costs, current estimates of program baselines, and the status of exit criteria. They present 
the projected total costs and quantities for all remaining years of an acquisition program’s 
life. DAES information is designed to provide indications of both potential and actual 
program problems to the USD(AT&L) and to the Assistant Secretary of Defense 
(Networks & Information Integration) before they become significant. The reports 
provide action taken or planned to mitigate future program problems (DoD, 2011). 
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H. SUMMARY 


In this chapter, we summarized the background information required to 
understand the scope of this research. We presented the three support systems that form 
the DoD’s acquisition system, and we briefly explained the federal contract types 
applicable to commercial acquisitions and the preferred contract types for each DoD 
acquisition phase. We concluded the chapter with the importance of R&D to acquisition 
program success and discussed the primary documents from which this study draws. 
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III. PURPOSE 


A. OVERVIEW 

The purpose of this research is to better understand indicators of program and 
contract cost variances. Establishing accurate cost information prior to entering the EMD 
and production phases is extremely important to ensuring a successful program (Carter, 
2011). We propose that preproduction and contract-level variances, including those 
reported during the EMD phase, are one possible set of indicators of future program cost 
variances. Interpreting these variances requires an understanding of the reasoning behind 
them and of their methods of construction. 

In this chapter, we provide the information required to understand program- and 
contract-level variances, including program cost, schedule, and engineering variances, as 
well as contract earned value cost and schedule variances. Previous research has been 
performed on cost overruns by contract type (Berteau et al., 2011), and recent initiatives 
have recommended the increased use of fixed-price contracts (Carter, 2010). One 
purpose of this research is to identify the effect of fixed-price contracts on program 
variances. 

In this chapter, we explain the regulations that govern these variances and that 
inform the decision-making of acquisition professionals. We also define each of the 
program and contract variances referenced in this research. We conclude the chapter 
with an explanation of why the study of these acquisition variances is important. 

B. ACQUISITION OVERSIGHT 

The defense acquisition process includes extensive program oversight and data 
reporting requirements. Program oversight has not produced the desired effect—capable 
systems delivered on-time and within budget. However, a closer review of the program 
and contract variance data collected under present reporting requirements may reveal 
novel conclusions about the interdependent nature of the problems associated with 
defense acquisition. 
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The current system of acquisition oversight is the result of numerous reforms 
(Rothenflue & Kwolek, 2010). In the following sections, we discuss some of these 
reforms, including the Nunn-McCurdy Amendment and several other influential reforms, 
which substantially contributed to the structure and content of the variance data analyzed 
in this research. 

1. Nunn-McCurdy Amendment 

One reform that significantly changed oversight procedures and focused on 
controlling cost growth was the Nunn-McCurdy Amendment. Nunn-McCurdy, first 
introduced in the Department of Defense Authorization Act of 1982, expanded the SAR 
requirements established in 1969 (Rothenflue & Kwolek, 2010). Nunn-McCurdy 
required program managers to submit SARs to Congress annually or immediately 
following a growth of 15% over the total program acquisition unit cost (PAUC) and 
average procurement unit cost (APUC). Nunn-McCurdy is still a public law, so program 
managers are still required to submit SARs. SARs must include any change in schedule 
milestones and system performance (Department of Defense Authorization Act, 1982). 
Additional reporting guidance has been added to the Nunn-McCurdy statute over the 
years and that guidance remains today as a control method for holding program managers 
accountable for cost growth on MDAPs. 

The most recent changes to the statute occurred in 2006 and 2009. The National 
Defense Authorization Act for Fiscal Year 2006 amended Nunn-McCurdy to include 
significant and critical dollar amount thresholds rather than a single threshold (GAO, 
2011). In the act, significant cost growth is defined as a 15% increase to the current 
baseline or a 30% increase to the original baseline for the PAUC or the APUC.i 
Significant cost growth requires congressional reporting. Critical cost growth is defined 
as a 25% increase to the current baseline estimate or a 50% increase over the original 
baseline estimate for the PAUC or APUC (GAO, 2011). 


1 PAUC = (Total Development $ + Procurement $ + Construction $)/Total Program Quantity. APUC = 
Total Procurement $/Procurement Quantity 
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It is important to understand estimates and baselines when discussing the Nunn- 
McCurdy Amendment. The current baseline estimate is defined as the latest estimate on 
an approved program, defined as the currently approved acquisition program baseline 
(APB). The original baseline estimate is defined as the APB approved at MS B or 
program initiation, whichever occurs later (Axtell & Irby, 2007). Each military Service 
must establish a baseline for each of its MDAPs, including parameters to describe the 
cost estimate (referred to as the baseline estimate), schedule estimate, performance 
estimate, and any other important factors of a MDAP (10 U.S.C. § 2435, 2011). 
According to 10 U.S.C. § 2435 (2011), the revision of the original baseline should be 
changed to the new baseline only after a critical (Nunn-McCurdy) breach; if this happens, 
the Secretary of Defense notifies Congress of the breach in the next SAR and gives 
reasons for the adjustment or revision (10 U.S.C. § 2435, 2011). 

The Weapon Systems Acquisition Reform Act of 2009 (WSARA) added further 
repercussions for programs with critical cost growth. Among other requirements, 
WSARA added the presumption that a program will be cancelled unless the Secretary of 
Defense “certifies (with reasons and supporting documentation) that continuing the 
program is essential to national security and that the program can be modified to proceed 
in a cost-effective manner” (Lymon, McWhorter, & Violette, 2011, p. 20). Critical cost 
growth also requires the program to “receive a new milestone approval (and associated 
certification) prior to the award of any new contract or contract modification extending 
the scope of the program” (Lymon et al., 2011, p. 20). 

WSARA also requires departments to perform an independent cost estimate 
supporting a program’s cost reasonableness and a stated confidence level for that estimate 
(Levin, 2009). The requirement for a stated confidence level echoes an observation made 
by the GAO (2008) in a report to the Senate Armed Service Committee: “To make more 
informed investment decisions, cost estimating best practices call for estimating a range 
of possible costs around a point estimate to provide information about the levels of 
uncertainty and confidence” (p. 24). There were seven Nunn-McCurdy breaches in 2010 
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to the APBs—four significant and three critical (USD[AT&L], 2011). In the following 
section, we explain APBs, the initial program baselines that Nunn-McCurdy breaches 
were based on. 

2. Acquisition Program Baseline 

The APB was established by the Defense Acquisition Improvement Act of 1986 
to create a baseline to improve program stability. The APB is defined by the DAU as the 
“baseline that reflects the threshold and objective values for the minimum number of 
cost, schedule, and performance attributes (called ‘key performance parameters’) that 
describe the program over its life cycle” (“Acquisition Program Baseline,” 2009). The 
APB answers how the system is supposed to perform, when critical events occur, and 
how much the program should cost. Every program manager must submit and receive 
approval for program goals prior to initiation of all acquisitions (USD[AT&L], 2008). 
The APB satisfies the requirement for goals on all ACAT I programs. 

By tracking actual program performance against established baselines, the 
program manager is alerted to potential problems and can take early corrective action. If 
a program breaches an approved baseline threshold, the program manager must submit a 
formal memo to the MDA and to the component’s leadership. A breach of performance 
is defined as a failure to meet the specific parameter’s threshold value as laid out in the 
APB. A breach of schedule is failure to meet the objective date plus six months. A 
cumulative program cost increase of 10% or greater from the approved cost baseline is a 
cost breach (“Acquisition Program Baseline,” 2009). 

3. Selected Acquisition Reports 

SAR requirements are covered under 10 U.S.C. § 2432 (2011). SAR submittal is 
required annually on December 31 unless a significant or critical cost growth occurred or 
a delay of six months in any current estimated milestone occurs, both of which require a 
quarterly report (10 U.S.C. § 2432, 2011). SARs contain the following 19 sections: 

1. Program Identification, 

2. Program Information, 
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3. Responsible Office, 

4. References, 

5. Mission and Description, 

6. Executive Summary, 

7. Threshold Breaches, 

8. Schedule, 

9. Performance, 

10. Track to Budget, 

11. Cost and Funding, 

12. Low-Rate Initial Production, 

13. Nuclear Cost, 

14. Foreign Military Sales, 

15. Unit Cost, 

16. Cost Variance, 

17. Contracts, 

18. Deliveries and Expenditures, and 

19. Operating and Support Cost. 

Section 16, Cost Variance, is reported in a SAR following four steps. In the first 
step, the total variance for each program’s appropriations estimate is calculated. Second, 
the category for the particular change is identified. Third, the dollar amount for each 
variance category by FY is determined. Finally, a clear and understandable explanation 
for the changes is provided (DoD, 2010). 

A program’s total CV comes from many different sources. These are aggregated 
in SARs into seven categories. In later chapters of this research, we describe the 
relationships deduced from SAR CV data and other data. Accordingly, it is important to 
understand the categories comprising the program’s total CV, which are shown in Figure 

7. 
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Figure 7. Total Program Variance Factors 
(DoD, 2010) 

The various cost variances are calculated for the SARs, which are uploaded to DAMIRS. 
The variance that occurred between reports for each factor in Figure 7 must be verbally 
explained in the SAR. 


According to a GAO (2011) study, engineering and design issues, schedule issues, 
and quantity changes were the primary reasons cited for unit cost growth that led to 
Nunn-McCurdy breaches. In a study performed by David McNicol (2004) on cost 
growth, he identified three areas that caused growth in procurement costs: increased 
system capabilities, an unrealistic estimate of cost growth, and poor program execution or 
exceptional budget instability. In this research, we focus on CV along with SV and EV 
as potential leading indicators of relative program success or failure since the other 
primary reason—quantity change—is more likely to be a lagging rather than leading 
indicator (Gansler, 2010). 
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4. Earned Value Management 

Former USD(AT&L) Dr. Jack Gansler signed a memorandum in August 1999 
announcing the DoD’s adoption of the earned value management system (Office of the 
Secretary of Defense, 2002). Earned value management (EVM) is a mandatory reporting 
requirement for cost and incentive contracts valued at $20 million or greater, to include 
all MDAPs, and is governed by DEARS (2011) subpart 234.2 and DoD Instruction 
5000.02 (USD[AT&L], 2008). The use of EVM is optional for the program manager on 
any contract valued at less than $20 million based on a cost-benefit analysis 
(USD[AT&L], 2008). 

EVM is a tool used to manage programs that attempts to integrate contract cost, 
schedule, and technical parameters in order to hold all parties accountable for large, 
complex acquisitions. Program managers are ultimately accountable for confirming that 
EVM requirements are included in statements of work, and program managers can utilize 
the variance measurements to forecast contract cost and schedule performance. The 
program manager can use EVM to track the status of the contracts within his or her 
program and use the measurements from EVM to build corrective action plans to get the 
program back on track. 

Problems with individual contracts may impede the progress of a portion of a 
program, or they may cause cascading problems across multiple contracts within a 
program and the program as a whole. In later chapters, we examine contract cost and 
schedule variances as determinants of relative program success. In the following section, 
we explain the main SAR output of EVM, contract cost and schedule variances, in greater 
detail. For the sake of clarity, in this research we avoid referring to earned value as “EV” 
in order to better distinguish earned value from engineering cost variance (EV), which we 
define later in this section. 

C. CONTRACT VARIANCES 

Two main EVM metrics are reported for every active MDAP contract, meaning 
every MDAP contract that is less than 90% complete: cost variance and schedule 
variance. Multiple cost and schedule contract variances are routinely reported on a single 
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program SAR. To better understand contract cost variance and schedule variance, we 
further define these terms in the next sections. 

1. Contract Cost Variance 

Contract cost variance is a metric that calculates cost performance by subtracting 
the actual cost of work performed (ACWP) from the budgeted cost of work performed 
(BCWP). The value is at a particular point of time and shows cumulative cost variance to 
date. A positive value is favorable because it indicates the work was performed under 
budget. A negative value is unfavorable because it indicates more money was spent than 
was budgeted for the task. 

Contracts experiencing unfavorable cost variance are likely to experience a 
contract cost overrun because it is difficult to reduce budgeted future work unless 
quantities or scope of work are reduced (“Cost Variance,” 2009). SAR contract cost 
variances are reported as the change in BCWP minus ACWP rather than in current year 
or base year dollars. For the sake of clarity, in this research we avoid abbreviating 
contract cost variance as “CV” in order to better distinguish contract cost variance from 
program cost variance (CV), which we define later in this section. 

2. Contract Schedule Variance 

Contract schedule variance is a metric that calculates schedule performance by 
subtracting the budgeted cost of work scheduled (BCWS) from the BCWP. The value is 
expressed for a specific period of time or is cumulative to date. A positive value is 
favorable because it indicates more work has been completed than scheduled. A negative 
value is unfavorable because it indicates that planned work was not completed. Both a 
slip in schedule and a failure to achieve certain technical milestones as planned can result 
in a negative schedule variance. 

Programs experiencing unfavorable schedule variance may also experience a 
delayed completion, but they can possibly recover in the future (“Schedule Variance,” 
2009). For the sake of clarity, in this research we avoid abbreviating contract schedule 
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variance as “SV” in order to better distinguish contract schedule variance from program 
schedule variance (SV), which we define later in this section. 

D. PROGRAM VARIANCES 

Over the last 20 years, the Army cancelled 22 major programs before they entered 
production, the costs already incurred were approximately $1 billion in 1996 and reached 
as high of $3.8 billion per year after 2004 (Ewing, 2011). Common eauses for program 
eaneellation are cost overruns, schedule slips, and underestimation or failure to eliminate 
technological risk. In this research, we examine at the program level three corresponding 
categories of variance (cost, schedule, and engineering variance) found in the SARs. To 
better understand the difference between these program varianees, we further define them 
in the next seetions. 

1. Cost Variance 

Program cost variance (CV) is reported in two general forms: as the cost change 
between the program’s current SAR and previous SAR whieh is the indicator we employ. 
In addition the cost change between the program’s current SAR and SAR baseline is also 
reported. A positive value is unfavorable because it indicates that the estimated cost of 
the program has inereased. A negative value is favorable beeause it indicates that the 
estimated eost of the program has deereased. Changes in CV are subdivided in seven 
categories: eeonomic, quantity, schedule, engineering, estimating, other, and support. 
The variances associated with these seven categories sum to the cost variances. We 
examine two of these categories, schedule and engineering, more closely in the following 
sections. 

2. Schedule Variance 

Schedule variance (SV) is cost variance attributable to schedule changes. SV is a 
eomponent of CV. Since SV is a component of CV, the two will eovary. Accordingly, 
the CV we used in this researeh provides explieit explanatory information about the 
subcomponents SV and EV. 


43 



3. Engineering Variance 

Engineering variance (EV) is cost variance attributable to engineering changes 
and is a component of program cost variance. EV is more difficult to quantify than SV 
and is referred to in DoD contracting offices as the cost of an approved engineering 
change proposal. This variance occurs due to new technology upgrades, redesign, and 
configuration changes. As with all SAR cost variance categories, a descriptive 
explanation is provided in the SAR to aid understanding of what caused the variance. 
Changes in support items are not included in EV (DoD, 2010). 

Engineering changes typically occur to a specific item identified on the work 
breakdown structure (WBS). A WBS defines the deliverable element by the scope of 
work. If the scope changes, an engineering variance to total cost will likely occur. 
Minimizing changes and beginning a program with a clear and well-defined requirement 
is the best prevention for poor engineering variance performance (GAO, 2011). 

Because engineering cost variance is a component of total program cost variance, 
the presence of any engineering cost variance will cause the two to display covariance. 

As a result, when we discuss CV it encompasses SV and EV. Because of this, when 
conducting much of the analysis, SV and EV are subtracted from CV for analysis in this 
paper for many of he models developed. We call this Program Net Cost Variance later in 
the analysis. 

E. REASONS TO STUDY 

Understanding what variables affect program costs and CVs, and the linkages 
between contract and program data are the purpose of this research. To better understand 
possible drivers of program CVs, we examine the historical effects of preproduction 
decisions on production CV outcomes. Improving program cost outcomes has been the 
subject of many acquisition reforms, but these reforms have not addressed the complexity 
and interdependence found in DoD acquisition programs that drive poor program cost, 
schedule, and technical outcomes (Rothenflue & Kwolek, 2010). 

We examine several potential cost variance drivers in later chapters, including 

contract type during the EMD acquisition phase, program and contract variances, and 
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MDAP type. It would be beneficial to the acquisition workforce to better understand 
each of these potential causes of variance. 

1. R&D’s Effect on Life-Cycle Costs 

R&D contracts for FY2010 totaled $80 billion (USD Comptroller, 2011). 
Increasing the number of R&D contracts issued on a fixed-price incentive basis could 
yield significant savings based on the presumption that fixed-price incentive contracts 
decrease cost overruns (Kendall, 2011). At one time, the DoD attempted to impose 
fixed-price incentive contracts on efforts in which significant invention could be 
anticipated, although recently the use of cost-plus-award-fee contracts has become 
widespread (Carter, 2010; Darst & Roberts, 2010). Under Secretary Carter (2010) has 
advocated the use of fixed-price incentive firm target contracts in the place of cost-plus- 
award-fee contracts wherever practicable, including EMD contracts which are of 
particular interest to policy-makers (Carter, 2011). 

The changing political favorability of particular contract types must be evaluated 
with the regulations implicit in the 13 federal contract types listed in FAR part 16; 
different contract types are better suited to different types of work. This truth is 
consistent with the discretion given to the acquisition team in FAR subpart 1.102 to “use 
sound business judgment” (FAR, 2011). The contract types most significant to MDAPs 
were discussed in Chapter II. 

Program managers and contracting officers should understand that the selection of 
an inappropriate contract type during EMD can negatively impact contract performance 
(Sadeh et al., 2000). Restricting R&D work to a fixed-price incentive basis could 
negatively affect problem detection and problem-solving early in a program if the risk 
conditions are such that a fixed-price contract type is not appropriate. Contracted 
companies could avoid researching all problems and alternative program solutions in 
order to increase their profit margin on the fixed-price contracts, provided that the 
contract specification requirements are satisfied. Poor problem detection early in a 
program will likely increase program costs in the long-term. Good EMD contract 
performance and appropriate problem-solving can address potential technical and system 
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integration issues before production, when they are most costly to address. Studying the 
effect of EMD contract type may give important insight into the wisdom of incentivizing 
program managers to select more restrictive contract types. 

2. Aid Current Practitioners’ Programmatic Decision-Making 

Examining potential sources of variance may improve the program managers’ 
understanding of reasons for the relative success of programs. A better understanding of 
these reasons could aid the decision-making of current practitioners. Preproduction 
program cost, schedule, and engineering variance and current period contract earned 
value cost and schedule variance are proposed by acquisition managers as possible 
indicators of future program cost variances. CV, SV, and EV within individual programs 
have been qualitatively linked and, to an extent, quantitatively linked (Rothenflue & 
Kwolek, 2010). Eurther study is necessary to better determine the nature of the 
interdependence between these program and contract variances. An improved 
understanding of this interdependence could enhance practitioners’ ability to estimate a 
contract’s most likely final cost, a program’s schedule or cost variances, or a program’s 
relative cost risk, given past program and contract variances. 

Other factors are proposed by acquisition managers as possible reasons for 
program cost variances, including MDAP segment. An MDAP segment is the acquisition 
portfolio segment that a program falls into, such as aircraft, missiles, ammunition, 
shipbuilding, or other, as defined by the categories listed in the National Defense Budget 
Estimates for FY2012. In this research, we do not further differentiate between these 
categories, although such differentiations can be found in the Air Eorce’s Appropriation 
Symbols and Budget Codes (Fiscal Year 2012), in order to permit program comparisons 
between similar programs across Services (Department of the Air Force, 2011). Some 
studies of cost and schedule variances have been restricted to a particular MDAP 
segment, such as aircraft (Rothenflue & Kwolek, 2010); such studies inherently exclude 
the possible effects of the MDAP segment. Constraining a study to a particular contract 
type or MDAP segment can control for effects associated with details of that specific type 
of contract. Although it is unlikely that policy-makers would develop fewer programs of 
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a given MDAP segment simply due to the relative success of that type of program, 
examining the relative success of the various MDAP categories in conjunction with other 
factors could improve acquisition professionals’ understanding of the relative risk present 
in each MDAP segment. 

Improving practitioners’ understanding of the risks present in MDAPs based on 
past program performance may enhance future program outcomes by aiding cost 
estimating and improving program risk assessment. Such developments could improve 
program outcomes by reducing funding volatility due to cost overruns and could decrease 
technical risk by improving risk assessment. Other improvements are possible, including 
program improvements driven by further reform of the acquisition system such as setting 
shorter program timelines, promoting real competition, and increasing the use of fixed- 
price incentive contracts (Carter, 2010). 

3. Recommended Future R&D Management Reforms 

A better understanding of cost and schedule variances could identify areas of 
current policy that would benefit from reform. Novel findings regarding any of the 
researched causes of program variances could yield potential reform improvements. 
Conversely, research conducted on the effects of acquisition reforms on defense 
programs could provide direction for creating more effective policies (Searle, 1997). 

Relating the efficacy of past reforms through a review of various yearly effects on 
variances over time could permit a qualitative review of past reform. Two such studies 
(Christensen, Searle, & Vickery, 1999; Drezner, Jarvaise, Hess, Hough, & Norton, 1993) 
showed that program reform initiatives did not improve program performance and, in 
fact, cost growth worsened after initiatives were implemented. It could be useful to 
question the conventional thinking that past reforms such as the Federal Acquisition 
Strea ml ining Act (FASA) of 1994 have had a negligible effect on MDAP outcomes 
(Smirnoff & Hicks, 2008). The most recent acquisition reforms, such as WSARA, have 
generally focused on reducing risk. It is possible that such reforms may already be 
improving the acquisition process. 
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F. SUMMARY 

The purpose of this research is to better understand the determinants program and 
contract cost and cost variances. Each possible cause of program variance discussed in 
this chapter is worthy of further research in order to understand the possible causes of 
program cost variances. The various program cost variances and contract variances we 
discussed in this chapter can be qualitatively linked to cost, schedule, and technical 
performance. We explore these interdependent variables and other possible causes of 
variance quantitatively in Chapter IV. 
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IV. DATABASE CONSTRUCTION AND ANALYSIS 


A. OVERVIEW 

In this chapter, we provide an overview of the data we used to analyze the effect 
of program and contract variances, program segment, contact type and other variables on 
program and contract cost during the systems acquisition phase of MDAPs. In this 
analysis, we seek to close the existing gaps in the understanding of program and contract 
management data. Cross-referencing contract variances with program variances provides 
acquisition professionals a more complete picture of program and contract changes that 
occur during both the EMD and production phase of the acquisition process. 

We first discuss the cross-sectional, time-series data we collected from available 
resources. The collection of this data permitted us to construct a database containing the 
numerical and categorical data that we used in the descriptive statistical analysis in this 
chapter. We provide the descriptive statistics to explore the initial data and to identify 
initial patterns among programs and contracts. These descriptive statistics enhance 
understanding of the data we employed in the formal empirical analysis contained in 
Chapter V, in which exploratory empirical models are estimated. 

We then provide a brief overview of cross-sectional, time-series analysis for 
reader understanding of the models that we estimated. Finally, we present a simple flow 
diagram depicting the relationships. Our objective is to determine whether the data are 
consistent so that, in turn, we could determine the effects of contract types and variances 
on program variances, and if the data are not consistent, based on statistical analysis, an 
alternative exploratory model is developed. We present this alternative structure and the 
final conclusions of our analysis in Chapter VI. 

B. DATA COLLECTION 

In this research, we used quantitative analysis to answer the research questions. 

Quantitative studies require that a number of assumptions be made. These include the 

assumptions that the process used is statistically reliable and operationally meaningful 

and that legitimate generalizations can be made from the sample to predict, explain, and 
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understand the population (Creswell, 1994). The method we used was to collect and 
model our data in the most unbiased manner possible. We also examined normality and 
other characteristics of the data. In our research, the population was all MDAPs and the 
sample was the dataset available that fit the required analysis techniques we applied in 
order to objectively make observations and generalizations about MDAPs. 

Our analysis in this study relied on data contained in DAMIRS and FPDS. 
Because each of the Services use these databases to document acquisition information, 
the extensive number of SARs provided the most meaningful data. We downloaded each 
program SAR from DAMIRS and transferred the SAR into Excel, breaking the 
information in the SARs down by program and contract to allow us to analyze the 
program and contract variables. When multiple contract line item numbers (CLINs) that 
are reportable in a SAR were present on a contract, we recorded each one in the database 
as a separate contract observation, in keeping with the practice used for SARs. We 
conducted further data collection in FPDS to determine predominant contract type and 
obtain data missing in the SARs. Figure 8 shows the number and percentage of the 
different contract types. 
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Figure 8. Actual Contract Type Breakdown 

We grouped contract type into six categories: FFP, FPIF, CPIF, CPAF, CPFF, and 
Hybrid/Other. The Other category only contained 5% of the contracts and included FPIS, 
FPEPA, Time and Materials, hybrid contract types with a 50-50 split in contract type, and 
any indeterminate predominant contract type. A 50-50 split in contract type was found in 
the database for one contract; since this contract was shown as a perfect split between 
CPAF and CPIF in the SARs, determining a predominant contract type was not possible. 
Several indeterminate predominant contract types were also observed; these were most 
common when a contract had been awarded, but not yet definitized. Some undefinitized 
contracts in the data could be the result of the program office and contractor still 
negotiating predominant contract type, particularly in the case of letter contracts. 
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We also broadly grouped contract type into cost-plus and fixed-price to compare 
these two categories. The exact number of each contract type for each program can be 
seen in Table 2, along with the total number of contracts for each program. Table 2 also 
includes the predominant contract type in EMD, RDT&E, and production. We 
determined predominant contract type based on the total current contract price for each 
program. We summed the total, current contract price for each category for the EMD and 
the Production phases, and for RDT&E appropriation contracts. The contract type 
associated with the largest total, current contract price was designated as the predominant 
contract type. The EMD predominant contract type variable lacked variability, since no 
program had a predominant fixed price orientation during EMD. Due to a lack of 
variability in predominant contract type during the EMD phase, we extracted the 
predominant contract type associated with RDT&E appropriations and treated RDT&E as 
a proxy for EMD, recognizing that using a proxy limited our ability to generalize and 
draw substantial conclusions from our results. Using a proxy for predominant EMD 
contract type was thought to be better than not attempting to model predominant EMD 
contract type at all. 
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Table 2. MDAP Number of Contracts, Predominant Contract Type During EMD 

and Production, and RDT&E Appropriation 



Contract Type 

Total 

Predominant Contract Type 

Program 

CPAF 

CPFF 

CPIF 

FFP 

FPIF 

other 

EMD 

RDT&E 

Prod. 

AB3A 

0 

0 

12 

0 

0 

0 

12 

CPIF 

CPIF 

CPIF 

AEHF 

17 

0 

0 

0 

0 

0 

17 

CPAF 

CPAF 

CPAF 

AGM-88E AARGM 

0 

0 

7 

2 

2 

0 

11 

CPIF 

CPIF 

CPIF 

AIM-9X 

0 

5 

6 

18 

0 

0 

29 

CPIF 

CPIF 

FFP 

B-2 RMP 

0 

0 

2 

0 

2 

4 

8 

CPIF 

other 

FPIF 

C-130 AMP 

7 

0 

0 

2 

0 

0 

9 

CPAF 

CPAF 

FFP 

C-5 RERP 

3 

0 

0 

6 

0 

14 

23 

CPAF 

CPAF 

Other 

CEC 

27 

6 

0 

36 

1 

0 

70 

CPAF 

CPAF 

CPAF 

CH-47F 

0 

9 

7 

15 

7 

0 

38 

CPIF 

FFP 

FFP 

E-2D ARE 

7 

0 

3 

3 

4 

0 

17 

CPAF 

CPAF 

CPAF 

EA-18G 

7 

4 

3 

2 

3 

12 

31 

Other 

CPAF 

FPIF 

EXCALIBUR 

0 

0 

6 

8 

0 

0 

14 

Other 

CPIF 

CPIF 

F-22 

13 

9 

1 

47 

0 

9 

79 

CPAF 

CPAF 

FFP 

FBCB2 

8 

0 

3 

8 

5 

0 

24 

None 

CPAF 

FFP 

H-1 UPGRADES 

11 

2 

0 

20 

0 

0 

33 

CPAF 

CPAF 

FFP 

HIMARS 

0 

0 

0 

25 

0 

0 

25 

None 

None 

FFP 

JASSM 

12 

5 

6 

23 

2 

0 

48 

CPFF 

CPAF 

FFP 

JDAM 

4 

0 

0 

19 

0 

0 

23 

None 

CPAF 

FFP 

JSOW Baseline 

0 

0 

2 

5 

3 

0 

10 

None 

CPIF 

FFP 

LUH 

0 

0 

0 

5 

0 

0 

5 

None 

None 

FFP 

MH-60R 

0 

9 

15 

27 

2 

4 

57 

CPFF 

CPIF 

FFP 

MH-60S 

0 

0 

18 

29 

0 

0 

47 

None 

CPIF 

FFP 

MUGS 

1 

0 

6 

0 

3 

1 

11 

CPIF 

CPIF 

CPIF 

NAS 

0 

0 

0 

13 

0 

0 

13 

None 

FFP 

FFP 

P-8A 

7 

0 

0 

2 

0 

0 

9 

CPAF 

CPAF 

CPAF 

PATRIOT PAC-3 

0 

0 

28 

23 

0 

4 

55 

CPIF 

CPIF 

FFP 

SM-6 

7 

0 

0 

0 

1 

0 

8 

CPAF 

CPAF 

CPAF 

SSN 774 

2 

25 

5 

0 

29 

0 

61 

None 

FPIF 

FPIF 

Tactical 

Tomahawk 

0 

5 

0 

15 

6 

0 

26 

CPFF 

CPFF 

FFP 

UH-60M 

8 

4 

0 

6 

0 

0 

18 

CPAF 

CPAF 

FFP 

V-22 

13 

2 

7 

38 

13 

0 

73 

None 

CPAF 

FPIF 

Total 

154 

85 

137 

397 

83 

48 

904 
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The major program information we extracted from the SARs included the DoD 
component, the milestones, the projected program, and the variances. The contract 
information included initial, current, and final price data and earned value variances. 
From this collection process, we built an extensive dataset. We developed additional 
variables to conduct further analysis. We categorized the programs into four segments: 
(1) Aircraft, including planes and helicopters; (2) Missile, weapons and ammunition; (3) 
C4ISR, including radar, satellite and communication systems; and (4) Ship and 
submarine. Figure 9 shows the breakdown with the number of programs in each 
segment. 



MDAP Segment 

Figure 9. Number of MDAPs by Segment 

The database consists of cross-sectional, time-series data. The cross-section 
includes multiple programs and their attendant contracts during a particular year. The 
time-series consists of individual programs and their attendant contracts identified by 
SAR data over a several-year period. A current list of active MDAPs consisted of 84 
programs. In order to perform a thorough analysis of the preproduction variance effect 
on production, we chose only those MDAPs that included pre- and post-MS B data. To 


54 










observe the variance effects over time, a minimum of a five-year span in data availability 
was necessary. This narrowed the number available programs to 31, listed in Table 3 
with their full popular name, branch of service, segment category, and type. The segment 
codes are: Aircraft = A; Missile, weapons, and ammunition = M; C4ISR = R; and Ship 
and submarine = S. The program type is either modification (Mod) or a new start (New). 
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Table 3. Programs Included in the Analysis 


Program 

Full Name 

Service 

Segment 

AB3A 

Longbow Apache - Block IMA Remanufacture 

Army 

A 

C-130 AMP 

C-130 Avionics Modernization Program 

Air Force 

A 

C-5 RERP 

C-5 Reliability Enhancement and Reengining 

Program 

Air Force 

A 

CH-47F 

CFI-47F Improved Cargo Flelicopter 

Army 

A 

E-2D ARE 

E-2 Advanced Hawkeye 

Navy 

A 

EA-18G 

EA-18G Growler 

Navy 

A 

F-22 

F-22 Advanced Tactical Fighter 

Air Force 

A 

H-1 

UPGRADES 

H-1 UPGRADES (4BW/4BN) 

Navy 

A 

LUH 

Light Utility Flelicopter 

Army 

A 

MH-60R 

MH-60R Multi-Mission Helicopter 

Navy 

A 

MH-60S 

MH-60S Fleet Combat Support Helicopter 

Navy 

A 

P-8A 

P-8A POSEIDON 

Navy 

A 

UH-60M 

UH-60M BLACK HAWK 

Army 

A 

V-22 

V-22 Joint Services Advanced Vertical Lift Aircraft - 
Osprey 

Navy 

A 

AGM-88E 

AARGM 

AGM-88E Advanced Anti-Radiation Guide Missile 

Navy 

M 

AIM-9X 

AIM-9X Air-to-Air Missile 

Navy 

M 

EXCALIBUR 

Excalibur Precision 155mm Projectiles 

Army 

M 

HIMARS 

High Mobility Artillery Rocket System 

Army 

M 

JASSM 

Joint Air-to-Surface Standoff Missile 

Air Force 

M 

JDAM 

Joint Direct Attack Munition 

Air Force 

M 

JSOW 

Baseline 

Joint Standoff Weapon Baseline Variant and Unitary 
Warhead Variant 

Navy 

M 

PATRIOT 

PAC-3 

Patriot Advanced Capability-3 

Army 


SM-6 

Standard Missile-6 

Navy 

M 

Tactical 

Tomahawk 

Tactical Tomahawk R/UGM-109E 

Navy 


AEHF 

Advanced Extremely High Frequency Satellite 

Air Force 

R 

B-2 RMP 

B-2 Radar Modernization Program 

Air Force 

R 

CEC 

Cooperative Engagement Capability 

Navy 

R 

FBCB2 

Force XXI Battle Command Brigade and Below 
Program 

Army 

R 

MUOS 

Mobile User Objective System 

Navy 

R 

NAS 

National Airspace System 

Air Force 

R 

SSN 774 

SSN 774 Virginia Class Submarine 

Navy 

s 


Type 

Mod 

Mod 

Mod 

Mod 

Mod 

New 

New 

Mod 

New 

New 

Mod 

New 

Mod 

New 

Mod 

New 

New 

New 

New 

New 

New 

New 

New 

New 

New 

Mod 

New 

New 

New 

New 

New 



























































































































































We performed normality tests on the variables in the dataset. There are a number 
of graphical and non-graphical normality tests. A simple graphical test of normality is a 
histogram. We graphed a histogram for each dependent variable to determine whether 
the variance measures were normally distributed. The variance measures that were the 
primary focus of the research each displayed a bell shape similar to a normal distribution, 
but with excessive kurtosis that caused the data to be non-normal. The statistical 
methods used in this research were sufficiently robust to accommodate the use of 
variables exhibiting non-normality. Further explanation of the non-normality exhibited 
can be found in Appendix B. 

C. DESCRIPTIVE STATISTICS 

The questions we address in this research required quantitative analysis. 
Typically, the first step undertaken in an empirical analysis is to establish descriptive 
statistics (Hair, Black, & Anderson, 2009). In this research, we examined two basic data 
types: nominal data and interval data. Each type of data required a different type of 
analysis. 

Interval data are the least restrictive of these two data types. Also known as scale 
data, interval data include observations that can be compared numerically. Program cost 
variances, schedule variances, and engineering variances, and contract cost variances and 
schedule variances are examples of interval data. The program variances are reported in 
constant 2010 millions of dollars, and have the same scale. The contract variances are 
reported in then-year millions of dollars over the course of the contract. Conversion of 
the contract variances to constant 2010 millions of dollars was not attempted, since 
measuring the period of performance and the weighted spend rate for observation would 
have been prohibitively complicated. Some of the most common measures used to 
examine interval data include the mean, range, and standard deviation. Statistical 
techniques valid for nominal data are also frequently used. 

Nominal data include observations by category that cannot be placed in a logical 
order. The type of contract used for an acquisition is an example of nominal data. 
Although contracts exhibit risk along a scale in a defined order, not all contract types are 
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practical for a given acquisition, so it is not universally valid to generalize that CPFF 
follows CPAF, which follows CPIF. This precludes the consideration of contracts as 
ordinal data. Nominal data are examined for mode and frequency of occurrence, and 
typically are used to construet eategorical variables (0-1) for each nominal category. For 
nominal data, cross tabulation tables can be a helpful descriptive statistic for comparing 
the nominal categories of one variable with the categories or other variables. An example 
of a crosstab table is Table 2 showing the number of eontraet types for each program. 

In this research, we examined data to determine the causes of program and 
contract cost growth. A critical first step in our statistical analysis was the application of 
descriptive statistics to the data. We examined nominal and interval data using 
appropriate statistical techniques. 

D. THEORETICAL EOUNDATION 

Following the completion of the first step of applying descriptive statistics, we 
used several statistical techniques to further examine the cross-sectional, time-series 
database we developed from DAMIRS. The primary technique we employed was 
multiple regression analysis. We used multiple regression analysis to model the 
association between each included explanatory variable and the dependent variable of our 
eross-seetional, time-series dataset. After eonducting initial regressions using a structure 
based on our aequisition experienee, we revised the models taking into aeeount both the 
statistical properties of the models and our operational knowledge of acquisition. 

The use of a cross-sectional, time-series analysis allowed us to introduce multiple 
observations of the acquisition program and the associated contracts into the complete 
dataset across the program reporting times identified in the SARs. In this type of cross- 
sectional, time-series analysis, the statistieal tests are identieal to those applicable in 
traditional regression analysis. Based on our experience, we felt that the use of fixed- 
price contracts during EMD would affect program outcomes; however, there were no 
programs in which fixed-price contracts were the predominant eontraet type during EMD. 
Because there were programs in which the predominant contract type was fixed-price 
during the production phase of the acquisition process, we examined this categorical 
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variable. We extracted the contract type data associated with RDT&E appropriations in 
order to compare it to the predominant contract type in the production phase. 


E. CONCEPTUAL MODELS 

As indicated, our objective in this research is to determine the likely effects of the 
increased usage of fixed-price contracts during the acquisition process on program and 
contract costs. In order to meet this objective, we designed an interrelationship model of 
the variables. Figure 10 contains the basic variables and displays the conceptual 
hypotheses of how contract type affects contract variances and, ultimately, how it affects 
program variances. These variances are measured in dollars over time, and the factors 
also correspond to the independent and dependent variables we examined in the cross- 
sectional, time-series analysis. 



Program 

Variances 


® Contract 
Variances 


^Contract 


Type 


Figure 10. Flow Diagram of Conceptual Hypothesis 
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F. SUMMARY 

In this chapter, we described our data collection process and displayed some key 
features of this data. We also provided a brief discussion of the statistical methods we 
used for our analysis. These methods included descriptive statistics and multiple 
regression analysis. In the next chapter we continue our discussion of methodology and 
report the results of the statistical analysis we conducted. 
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V. STATISTICAL METHODOLOGY AND RESULTS 


A. OVERVIEW 

In this chapter, we initially provide a discussion of regression analysis as it relates 
to our exploratory models. We then provide an explanation of the results from our 
analyses and answer the research questions we proposed in Chapter I. We identify the 
relationship between both program and contract cost and program and contract variances 
as programs progress through development to production. Understanding these 
interrelationships should help acquisition professionals better manage cost, schedule, and 
technical risk in MDAPs. 

B. MULTIPLE REGRESSION ANALYSIS 

The primary analytical tool used in this chapter is multiple regression analysis. 
This technique estimated the effect of a number of specified explanatory variables on a 
particular dependent variable. In this analysis, historical observational data are 
employed, and one can expect these variables to be correlated with each other. 

However, multiple regression analysis is designed to deal with this issue. When 
one of the explanatory variables changes, the others are statistically held constant so the 
effect of the changing explanatory variables on the dependent variables is isolated. 

It is important to appreciate that a multiple regression model with a low can 
still have important uses. When t-statistics are statistically significant at the same time as 
the is low the model can be used to accurately estimate the change in the dependent 
variables, when a particular policy variable changes. In this situation, we can have a 
valid policy response model. 

In forecasting models, the R^ typically needs to be large. However, it remains 
true that in policy response models with low R^ and high t-statistics, one can accurately 
predict the value of dependent variables for a group of cases with like values for the 
explanatory variables even though there may be significant uncertainty associated with 
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predicting for an individual case. It is frequently the case that one makes policy based on 
what is likely to happen on average (Glister, 1970). 

In this analysis, we also make extensive use of categorical variables that are 
formed from some group that take on a value of 1 if some condition holds, say, a specific 
type of contract, and 0 otherwise. In the case of contract type, as an example, one of the 
contract types is selected as the reference category, and excluded from the model. The 
size and significance of the group’s included explanatory variables is always evaluated 
relative to the excluded categorical variables. Therefore, if an included categorical 
variable is not statistically significant, this means that it is not significantly different from 
the excluded variable. 

It is our expectation that the current analysis will be followed by subsequent 
analyses, and that as result, a consensus may emerge as to the appropriate specification 
that accurately reflects the underlying causal structure. Then, it will be possible to 
determine whether the data are consistent with this causal structure. However, we view 
our regression analyses as exploratory in nature, and this, in turn, resulted in several 
decisions as to how to display the results. 

For example, when the binary variables that are members of a particular group are 
all non-significant, we frequently display the results obtained. While an alternative 
would be to simply state that none of the binary variables are significantly different from 
each other, including all of the non-significant binary variables may aid future analysts 
who build upon our work. 

Also, a variable believed to be important to the analysis that has the predicted 
sign, but which is statistically very insignificant, may be retained in the model. This can 
also aid future analysts who believe this variable to be important and choose to include it 
in a model with a different specification. 

We have also limited our exploratory examination to the estimation of a number 
of single-equation regression models. Linked regression model that contain more than a 
single equation are not included. For example, we did not evaluate an empirical path in 
which, say, contract schedule variance, and other variables affect, say, program 
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engineering variance, which in conjunction with other explanatory variables, affects net 
program variance. This type of path-analysis modeling awaits additional research. 

C. DEFINITION OF VARIABLES 

1. Types of Data 

We now provide more detail on the types of data employed. As discussed in 
Chapter 4, the variables used in this research can be grouped into two categories: 
numerical and nominal. Numerical data can be compared using regression and other 
means, but nominal data cannot be directly compared. Instead, nominal variables must 
be converted to a series of binary variables, which are also frequently called dummy 
variables. Accordingly, dummy variables used in the research end in a “_D.” 

2. Binary Variables 

As indicated above, binary variables indicate the presence or lack of presence of a 
category or condition. Two or more categories may be compared using dummy 
variables. For example, two types of acquisitions are considered in this research: new 
start acquisitions and modification acquisitions. Acquisition type is represented by the 
variables “AcqType_Mod_D” and “AcqType_New_D.” When the value of 
AcqType_Mod_D is 1, this indicates that the MDAP is a modification program; when the 
value is 0, the MDAP is a new start program. When then value of AcqType_New_D is 1, 
this indicates that the MDAP is a new start program. Because the sum of these two 
variables necessarily equations 1, one of the two would be selected for explicit use in the 
regression, and the other would be the reference variables. 

As indicated, when examining the effect of a dummy variable, one should keep in 
mind the variable’s reference. The reference for AcqType_Mod_D is AcqType_New_D. 
Therefore, for regressions including AcqType_Mod_D, the effect of a modification 
acquisition relative to the effect of a new start acquisition is shown as the regression 
coefficient. This coefficient measures the difference found in the data between when 
AcqType_Mod_D equals one and when it equals zero. 
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The remaining dummy, categorical, and scale variables mentioned in the model 
are listed below, broken down by type and program or contract relationship. 

a. Additional Program Variables 

EMD_D: 1 indicates the presence of the EMD phase; 0 indicates the 
presence of the Production phase. 

RDTE_D: 1 indicates the presence of an RDT&E appropriation; 0 
indicates another appropriation, typically Procurement. 

PROC_D: 1 indicates the presence of the Procurement phase; 0 indicates 
another appropriation, typically RDT&E. 

AcqSegment_Aircraft_D: 1 indicates the presence of the Aircraft (planes 
and helicopters) Acquisition Segment; 0 indicates otherwise. 

AcqSegment_C4ISR_D: 1 indicates the presence of the C4ISR 
Acquisition Segment; 0 indicates otherwise. 

AcqSegment_Missiles_D: 1 indicates the presence of the Missiles 
Acquisition Segment; 0 indicates otherwise. 

AcqSegment_Ships_D: 1 indicates the presence of the Ship Acquisition 
Segment; 0 indicates otherwise. 

EMD_FFP_D: 1 indicates FFP as the predominant contract type during 
EMD; 0 indicates otherwise. 

EMD_FPIF_D: 1 indicates FPIF as the predominant contract type during 
EMD; 0 indicates otherwise. 

EMD_CPIF_D: 1 indicates CPIF as the predominant contract type during 
EMD; 0 indicates otherwise. 

EMD_CPAF_D: 1 indicates CPAF as the predominant contract type 
during EMD; 0 indicates otherwise. 
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EMD_CPFF_D: 1 indicates CPFF as the predominant contract type during 
EMD; 0 indicates otherwise. 

EMD_CONTOTH_D: 1 indicates Other/Hybrid as the predominant 
contract type during EMD; 0 indicates otherwise. 

RDTE_FFP_D: 1 indicates FFP as the predominant contract type for 
RDT&E; 0 indicates otherwise. 

RDTE_FPIF_D: 1 indicates FPIF as the predominant contract type for 
RDT&E; 0 indicates otherwise. 

RDTE_CPIF_D: 1 indicates CPIF as the predominant contract type for 
RDT&E; 0 indicates otherwise. 

RDTE_CPAF_D: 1 indicates CPAF as the predominant contract type for 
RDT&E; 0 indicates otherwise. 

RDTE_CPFF_D: 1 indicates CPFF as the predominant contract type for 
RDT&E; 0 indicates otherwise. 

RDTE_CONTOTH_D: 1 indicates Other/Hybrid as the predominant 
contract type for RDT&E; 0 indicates otherwise. 

PROD_FFP_D: 1 indicates FFP as the predominant contract type during 
Production; 0 indicates otherwise. 

PROD_FPIF_D: 1 indicates FPIF as the predominant contract type during 
Production; 0 indicates otherwise. 

PROD_CPIF_D: 1 indicates CPIF as the predominant contract type during 
Production; 0 indicates otherwise. 

PROD_CPAF_D: I indicates CPAF as the predominant contract type 
during Production; 0 indicates otherwise. 

PROD_CPFF_D: I indicates CPFF as the predominant contract type 
during Production; 0 indicates otherwise. 
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PROD_CONTOTH_D: 1 indicates Other/Hybrid as the predominant 
contract type during Production; 0 indicates otherwise. 

EMD_BasieContractType_CP_D: 1 indieates a cost-plus basic contract 
type as the predominant basie eontraet type during EMD; 0 indieates otherwise. 

EMD_BasicContractType_FP_D: 1 indicates a. fixed-prices basic contract 
type as the predominant basic contract type during EMD; 0 indicates otherwise. 

RDTE_BasicContractType_CP_D: 1 indieates a cost-plus basic contract 
type as the predominant basie contract type for RDT&E; 0 indicates otherwise. 

RDTE_BasicContractType_FP_D: 1 indicates di fixed-prices basic contract 
type as the predominant basic contract type for RDT&E; 0 indicates otherwise. 

PROD_BasieContractType_CP_D: 1 indieates a cost-plus basie eontraet 
type as the predominant basie contract type during Production; 0 indicates otherwise. 

PROD_BasicContractType_FP_D: 1 indicates di fixed-prices basic contract 
type as the predominant basic contract type during Production; 0 indicates otherwise. 

b. Contract variables 

FFP_D: indicates a contract is FFP; 0 indicates otherwise. 

FPIF_D: indicates a contract is FPIF; 0 indicates otherwise. 

CPIF_D: indieates a eontraet is CPIF; 0 indieates otherwise. 

CPAF_D: indicates a eontraet is CPAF; 0 indicates otherwise. 

CPFF_D: indieates a contract is CPFF; 0 indicates otherwise. 

CONTOTH_D: indicates a contract is Other/Hybrid; 0 indicates 

otherwise. 

FFP_EMD_CaseContraetType_D: 1 indicates an FFP contract during 
EMD; 0 indicates otherwise. As sueh, the coeffieients of FFP_D and 
FFP_EMD_CaseContractType_D can be summed to determine the performance of FFP 
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during EMD. The remaining Case Contract Type variables should be interpreted in the 
same manner. 

FPIF_EMD_CaseContractType_D: 1 indicates an FPIF contract during 
EMD; 0 indicates otherwise. 

CPIF_EMD_CaseContractType_D: 1 indicates a CPIF contract during 
EMD; 0 indicates otherwise. 

CPAF_EMD_CaseContractType_D: 1 indicates a CPAF contract during 
EMD; 0 indicates otherwise. 

CPFF_EMD_CaseContractType_D: 1 indicates a CPFF contract during 
EMD; 0 indicates otherwise. 

CONTOTH_EMD_CaseContractType_D: 1 indicates an Other/Hybrid 
contract during EMD; 0 indicates otherwise. 

3. Categorical Variables without Dummy Equivalents 

a. Program Variable 

ProgramID: A unique Program Identification number associated with a 
contract representing a single scope of work. 

b. Contract Variable 

ProgramandContractID: A unique Contract Identification number 
associated with a contract representing a single scope of work. This Identification 
number is separate and distinct from a contract’s contract number. 

4. Scale Variables 

a. Program Variables 

MilestoneBscheduleachieved_Fractional_Year: The year in which 
Milestone B was achieved, represented numerically as whole years with decimal partial 
years. 
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Program_Fractional_Year: The number of years since formal program 
inception (typically at Milestone B), represented numerically as whole years with decimal 
partial years. 

SARBaselineProdEstConstant2010$M: The program baseline cost 
estimate. This variable and the remaining variables labeled Constant2010$M are 
presented in constant 2010 millions of dollars. 

Program Schedule Variance, Constant2010$M: The current period cost 
variance attributable to schedule changes. 

Program Engineering Variance, Constant2010$M: The current period cost 
variance attributable to engineering changes. 

SubtotalCurrentChangesConstant2010$M: The current total cost variance. 

CECostVarianceConstant2010$M: This is the expected program cost 
through the end of production, and is the sum of the program baseline cost estimate, prior 
cost variance, and current cost variance. 

Program Cost Variance, Constant2010$M: The current period cost 
variance attributable to any program change, with the exception of schedule and 
engineering changes. This variance is also presented in constant 2010 millions of dollars. 

CurrentLessInitialProgramCost: The current estimate of program cost 
minus the initial estimate of program cost, presented in constant 2010 millions of dollars. 

b. Contract Variables 

TargetInitialContractPrice$M: The initial target contract price target in 
millions of dollars. 

TargetCurrentContractPrice$M: The current target contract price in 
millions of dollars. 

ProgramManagerEstimatedPriceAtCompletion$M: The current program 
manager’s contract estimated price at completion in millions of dollars. 
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Contract Cost Variance: The contract earned value cost variance in 
millions of dollars. 

Contract Schedule Variance: The contract earned value schedule variance 
in millions of dollars. 

CurrentLessInitialContractPrice: The current estimate of contract price 
minus the initial estimate of contract price in millions of dollars. 

ContractEACLessInitialContractPrice: The current contract estimate at 
completion minus the initial estimate of contract price in millions of dollars. 

ContractEACLessCurrentContractPrice: The current contract estimate at 
completion minus the current estimate of contract price in millions of dollars. 

For an alphabetized list of the variables used see Appendix A. 

D. RESEARCH QUESTIONS 
1. Primary Questions 

1. What effect does fixed-price R&D have on cost, schedule, and technical 
performance? 

Table 2 in Chapter IV showed the predominant contract type computed for RDT&E 
appropriation. There were three programs (CH-47F, NAS, and SSN 774) with 
predominant fixed-price contracts for RDT&E. In order to answer this research question, 
we computed the average CV, SV, and EV for all 31 MDAPs in the sample to contrast 
the effects of predominant fixed-price RDT&E contracts on the three MDAPs. The 
averages are listed in Table 4. 
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Table 4. Average Program Variances for Entire Sample and Predominant FP in 

RDT&E (2010 $M) 



CV 

SV 

EV 

Average Program Variance all 
MDAPs 

3093.6 

366.5 

651.6 

Average Program Variance of 
Predominant FP Contracts in 

RDT&E 

9306.0 

614.0 

685.3 


Table 4 shows that the average CV for the entire sample of MDAPs was $3,093.6 
million, while the average CV for predominant fixed-price RDT&E contracts was $9,306 
million, or an average CV increase of over $6 billion. There were only three programs 
with predominant fixed-price RDT&E contracts, which do not make these observations 
compelling evidence. One would assume that fixed-price contracts would assist in 
controlling program costs, but the use of fixed-price contracts during RDT&E is 
associated with larger CV. 

Table 4 also identifies the average SV on predominant fixed-price RDT&E 
contracts to be approximately $250 million higher than the sample, and EV was only 
higher by approximately $35 million. These figures, although computed on three 
programs, show that the predominant use of FP contracts during RDT&E is associated 
with higher SV. 

2. Do different segments of MDAPs (e.g., fighters, tanks, missiles, satellites) 
exhibit differing cost and schedule growth? 

We categorized the 31 MDAPs into four segments: A, M, R, and S. These 
segments are defined in Table 5, with the number of programs for each category shown in 
the total column. 
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Table 5. MDAP Segment 


CODE 

SEGMENT CATEGORIES 

Total 

A 

Aircraft (plane, helicopter) 

14 

M 

Missile, weapons, ammunition 

10 

R 

C4ISR 

6 

s 

Ship, submarine 

1 



31 


We also categorized the MDAPs into two types: modernization and new start. 
There were 10 modernization and 21 new start programs. To calculate average cost 
growth for each segment, we calculated the total cost growth (including both cost 
overruns and cost underruns) by subtracting the original SAR baseline from the latest 
program estimate, dividing by the number of years, and then dividing by the original 
SAR baseline. 2 This formula is shown in Figure 11. 

(Latest Program Estimate - SAR Original Baseline) 

Number of SAR Years 

Average Cost =_ 

Growth % 

SAR Original Baseline 

Figure 11. Average Cost Growth Formula 

To calculate schedule growth over the course of a program for each segment, we 
divided the total schedule variance (including both schedule overruns and schedule 
underruns) by the current SAR baseline. This formula is shown in Figure 12. 

Total Schedule Variance 

Average Schedule = _ 

Growth % 

SAR Current Baseline 

Figure 12. Average Schedule Growth Formula 


2 The segment average cost growth was computed from the annual average growth 
percentage rate of each program. 
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We calculated each program’s cost and schedule growth and averaged this by 
segment, shown in Table 6 for easy comparison. The Virginia Class Submarine was the 
only program in the ship and submarine category, thus it should not be considered as an 
average of the segment. The other three categories have a similar average cost growth, 
with aircraft topping the segments with 10.3% growth. The average schedule growth 
between the three categories varied greatly, with aircraft having the least amount of 
growth and the C4ISR segment averaging 13.4% growth in schedule. 


Table 6. Average Program Cost and Schedule Growth by MDAP Segment 


#of 

MDAPs 

SEGMENT CATEGORIES 

Average Cost 
Growth 

Average Schedule 
Growth 

14 

Aircraft (plane, helicopter) 

10.33% 

1.57% 

10 

Missile, weapons, ammunition 

7.20% 

3.18% 

6 

C4ISR 

9.87% 

13.42% 

1 

Ship, submarine 

2.96% 

2.65% 


3. Does early cost, schedule, or engineering variance serve as a leading indicator 
of later period CV, SV, and/or EV in either EMD or post Milestone C? 

Examining this research question required the use of multiple regressions, which 
we discuss in further detail in Section D. The regressions applicable to the 
interrelationship of variances can be found at the end of Section D. 

2. Secondary Questions 

1. What portion of MDAPs have fixed-price incentive R&D contracts? 

Based on the 31 programs and 904 contracts in the dataset, there were 32 fixed- 
price contracts issued during EMD from a total of 369 RDT&E-appropriated CLINs. 
Less than 9% of the 369 RDT&E contracts were fixed-price. The initiative promoting 
fixed-price R&D contracts is rather new, and the small amount of FP contracts is not 
surprising. Future research is recommended to investigate the increase of FP contracts 
used for RDT&E appropriation. 
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2. Is there qualitative information to support the assertion that fixed-price 
contracts prevent problem identification in the development process? 

The discussion of contract types and preferred contract types by acquisition phase 
in Chapter II and the discussion of the importance of R&D outcomes in Chapter III is 
consistent with the assertion that fixed-price contracts may be associated with problem 
identification in the development process. 

3. If the effect of fixed-price R&D is measurable, are the variances larger with 
regard to cost, schedule, or engineering during EMD and production? 

This question can be addressed as a continuation of primary research question 1. 
The CV, SV, and EV for predominant fixed-price contracts under RDT&E were 
compared to the entire dataset. The average CV, SV, and EV for predominant FP 
contracts during production was calculated and compared to the dataset listed in Table 7. 

Table 7. Average Program Variances for Entire Sample and Predominant FP in 

RDT&E and Production (in $ millions) 



CV 

SV 

EV 

Average Variance all MDAPs 

3093.6 

366.5 

651.6 

Average Variance of Predominant FP 
Contracts in RDT&E 

9306.0 

614.0 

685.3 

Average Variance of Predominant FP 
Contracts in Production 

3889.9 

333.7 

842.9 


The average variance for predominant fixed-price contracts in production was 


very similar to the entire dataset except for engineering variance. The average EV 
increase of $191 million occurring on predominant fixed-price contracts during 
production phase could be attributable to increased technical risk experienced during the 
production phase. Even fixed-price contracts might not be able to hedge the technical 
risk. The ceiling price might provide incentives for a contract change to be approved. 
Engineering changes occur due to new technology upgrades, redesign, and configuration 
changes. Minimizing changes would be the best prevention of poor engineering 
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variances. Further analysis might relate this finding to the associated contract type used 
during the EMD phase. 4. Based on the results found in this research, can any definitive 
policy recommendations be made? 

Our discussion of the results and recommendations in Chapter VI answer this 
research question. 

E. MULTIPLE REGRESION ANALYSIS 

We eonducted a regression analysis to identify the relationship between eost and 
price changes at the program and contract levels. These regressions partially formed the 
basis of our responses earlier in this chapter to the research questions we posed in 
Chapter I. In this seetion, we also explore regression relationships in greater detail by 
focusing on establishing the statistieal validity of the data, given knowledge of the 
acquisition process, rather than on responding directly to any specific research question. 
This was done using the stepwise method favored by most statisticians for exploratory 
regressions (Hair, et al., 2009). Following the stepwise method, we began with initial 
regressions and iteratively improved these regressions until arriving at satisfaetory final 
regressions. Differing initial and final regressions were run based upon the relationships 
we were attempting to quantify. 

We focused our modeling efforts first on relating cost and price changes over time 
at the program and contract levels. Then we related eost, sehedule, and engineering 
variances at the program level and earned value eost and sehedule varianees at the 
contract level. The remaining variables were included because, based on our knowledge 
of the acquisition process, we believed that they could have an effect on the dependent 
variables. After, developing program and eontraet models, we examined models in 
whieh program data was related to eontraet data. 

1. Regressions of Cost and Price Changes at the Program and Contract 
Level 

We focused our modeling effort on relating cost and price changes over time at 

the program and contract level; this focus arose from the general trend in contract and 

program cost growth over time, as illustrated in Figure 13. Program cost estimates for a 
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given SAR estimate the cost of an MDAP through the end of Production. Contract cost 
estimates for a SAR estimate the cost of the contract through the end of the contract in 
two forms: the current target price and Estimate at Completion (EAC). The current target 
price is the current negotiated contract price. The EAC is an independent estimation of 
the final contract price, based upon the expected price of definitized work, undefinitized 
work, and contract overruns. Our research uses the government program manager’s 
EAC. 


Program Cost Behavior 

Current 
estimate at 
most recent 
SAR 



Contract Price Behavior 


EAC 



Figure 13. Typical Program and Contract Cost Growth Over Time 

This typical pattern suggests multiple avenues for studying the causes of program 
cost growth, added contract scope, contract overruns, and the interrelationships between 
each. For this research, program cost growth is defined as the current estimate at time, t 
minus the initial cost. To better explore the relationship between program cost growth 
and contract price growth, we first examined the relationship between program cost 
growth and contract growth using the same explanatory variables included in the program 
cost regression, but adding contract level variables. Acquisition segment, acquisition 
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type, predominant basic contract type during production, predominant basic contract type 
for RDT&E, the years since program inception, the year that a program achieved MS B, 
and acquisition phase were each included as variables that explain program cost growth. 
The initial program cost growth model examined is shown in Table 8. 


Table 8. Initial Regression of Program Cost Growth (Current Estimate - Initial 

Cost) 


Model 


Coefficient 

Std. Error 

t-statistic 

1 

(Constant) 

-35576.462 

123146.728 

-.289 


AcqSegment_Aircraft_D 

-16700.178 

1083.424 

-15.414 


AcqSegment_C4ISR_D 

-16792.475 

1048.513 

-16.016 


AcqSegment_Missiles_D 

-16228.618 

1071.707 

-15.143 


AcqType_Mod_D 

1741.443 

600.233 

2.901 


PROD_BasicContractType_FP_d 

-587.125 

654.329 

-.897 


RDTE_BasicContractType_FP_d 

-236.511 

800.872 

-.295 


P rog ram_F racti o nal_Y ear 

215.176 

55.684 

3.864 


MilestoneBscheduleachieved_Fractional_Year 

25.898 

61.468 

.421 


EMD_D 

-1688.308 

647.847 

-2.606 

Dependent variable: CurrentLessInitialProgramCost 

= .448 

N = 827 


After exploring multiple possible alterations to the model, we eliminated variables 
with non-significant t statistics, namely the predominant basic contract type for RDT&E 
and the year that a program achieved MS B. The resultant final regression, shown in 
Table 9, has significant t statistics and a slightly improved R^. 
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Table 9. Final Regression of Program Cost Growth (Current Estimate - Initial Cost) 


Model 


Coefficient 

Std. Error 

t-statistic 

1 

(Constant) 

15820.918 

887.255 

17.831 

AcqSegment_Aircraft_D 

-16458.915 

700.657 

-23.491 

AcqSegment C4ISR D 

-16371.119 

810.449 

-20.2 

Acq Seg m e nt_M i ss i I es_D 

-16499.36 

691.423 

-23.863 

AcqType_Mod_D 

1770.771 

482.723 

3.668 

PROD_BasicContractT ype_FP_d 

-256.852 

514.441 

-0.499 

P rog ram_F racti o nal_Year 

186.603 

41.862 

4.458 

EMD_D 

-1493.173 

598.052 

-2.497 

Dependent variable: CurrentLessInitialProgramCost 

= .449 

N = 879 


The dummy variables in the acquisition segment category should be viewed in 
light of their relationship to their reference dummy variable, the acquisition segment 
ships dummy. The modification acquisition type dummy should be viewed in light of its 
relationship to the to the new start acquisition type dummy. Therefore, the coefficients 
shown for each dummy should be viewed in light of their reference — not in absolute 
terms. From this regression, we can determine that the acquisition segment dummies are 
each significantly different from their reference, ships; each segment has program cost 
growth that is significantly less than ships in relative terms. We can also determine that 
modifications experience relatively higher cost growth than new start acquisitions. The 
regression also shows that programs that use a contract from the fixed-price family of 
contracts (FFP, FPIF, FPIS, or FPEPA) during production experience lower cost growth 
than those that do not, although not at a significant level. This variable is retained in the 
model as an aid to those doing follow-on work. The regression also shows that programs 
experience significantly lower cost growth during EMD than during production; this is 
even more significant given that the regression also shows that cost growth significantly 
increases as the time since a program has achieved MS B increases. 

The next model examined the causes of contract price growth. We defined 
contract price growth as the contract estimate at completion (EAC) minus the contract 
initial target price. To better explore the relationship between program cost growth and 
contract price growth, we examined the relationship between contract price growth and 
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the variables we intended to hold constant for our later regression of program cost growth 
on contract cost growth. Acquisition phase, predominant contract type during EMD, 
acquisition type, acquisition segment, predominant basic contract type for RDT&E, the 
year that a program achieved MS B, and the years since program inception were each 
used as variables for explaining contract price growth. The initial contract price-growth 
model is shown in Table 10. 


Table 10. Initial Regression of Contract Price Growth (EAC - Initial Target) 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

4877.025 

17424.792 

.280 


EMD_D 

163.065 

92.847 

1.756 


EMD_CPAF_D 

433.518 

69.075 

6.276 


EMD_CPIF_D 

153.400 

83.308 

1.841 


EMD_CPFF_D 

-229.607 

84.751 

-2.709 


AcqType_Mod_D 

-560.877 

88.288 

-6.353 


AcqSegment_Aircraft_D 

420.955 

115.949 

3.631 


AcqSegment_C4ISR_D 

-115.839 

124.353 

-.932 


AcqSegment_Missiles_D 

18.759 

117.219 

.160 


MilestoneBscheduleachieved_Fractional_Year 

-2.454 

8.707 

-.282 


Program_Fractional_Year 

10.664 

8.308 

1.284 

Dependent Variable: ContractEACLessInitialContractPrice 

R^ = .163 

N = 891 


After exploring multiple possible alterations to the model, we eliminated a 
variable with a non-significant t statistic, the variable representing the year that a program 
achieved MS B. We also substituted specific contract type for each contract for 
predominant contract type during EMD. The resultant final regression, shown in Table 
11, has an improved R^. 
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Table 11. Final Regression of Contract Price Growth (EAC - Initial Target) 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

132.035 

168.037 

.786 


AcqSegment_Aircraft_D 

426.880 

121.139 

3.524 


AcqSegment_C4ISR_D 

20.141 

127.703 

.158 


AcqSegment_Missiles_D 

67.671 

121.148 

.559 


AcqType_Mod_D 

-340.531 

74.202 

-4.589 


Program_Fractional_Year 

28.110 

6.725 

4.180 


EMD_D 

107.048 

91.870 

1.165 


CPIF_D 

-346.545 

126.150 

-2.747 


FFP_D 

-461.813 

116.671 

-3.958 


FPIF_D 

-295.845 

142.762 

-2.072 


CPAF_D 

252.499 

124.183 

2.033 


CPFF_D 

-402.172 

138.872 

-2.896 

Dependent Variable: ContractEACLessInitialContractPrice 

R^= .189 

N = 891 


The contract-type dummy variables should be viewed in light of their reference, 
namely, the contract-type dummy variable in the Other/Hybrid segment. The only 
contract type with more growth than the reference contract type was CPAF. The reasons 
for the relative price increase cannot be determined from Table 11. However, larger price 
growth indicates that either contract overruns or scope increases resulted in CPAF 
contracts exhibiting greater price growth. This effect is still relative to that displayed by 
the reference contract type, Other/Hybrid. A similar interpretation applies to the 
Acquisition segment variables. 

In the next model we examined the causes of program cost growth, but this model 
also included contract price growth as an independent variable. This provides a linkage 
between the program data and the contract data. Again, we defined program cost growth 
as current cost minus initial cost. To better understand the output of this combined 
program cost and contract price growth model, we first ran the regressions above to 
understand what variables to include in the combined regression. Predominant basic 
contract type during production, predominant basic contract type for RDT&E, acquisition 
phase, contract type, acquisition type, acquisition segment, the years since program 
inception, and the year that a program achieved MS B were each included in the initial 
model. The initial program cost model, including the independent variable for contract 
price growth, is shown in Table 12. 
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Table 12. Initial Regression of Program Cost Growth (Current Estimate - Initial 
Cost), Ineluding the Independent Variable for Contraet Priee Growth 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

87866.712 

125688.232 

.699 


PRO D_Bas icCo ntractT y pe_F P_d 

-577.344 

650.660 

-.887 


R DT E_Bas icCo ntractT y pe_F P_d 

85.932 

793.588 

.108 


EMD_D 

-1865.279 

651.506 

-2.863 


CPIF_D 

-474.264 

1038.975 

-.456 


FFP_D 

-2526.073 

972.139 

-2.598 


FPIF_D 

-1150.724 

1099.022 

-1.047 


CPAF_D 

-1316.971 

1015.765 

-1.297 


CPFF_D 

-3715.515 

1088.713 

-3.413 


AcqType_Mod_D 

1311.159 

615.329 

2.131 


AcqSegment_C4ISR_D 

16464.854 

1069.747 

-15.391 


AcqSegment_Missiles_D 

16054.011 

1084.695 

-14.800 


Acq Seg m e nt_Ai rcraf t_D 

15935.673 

1095.288 

-14.549 


Program_Fractional_Year 

222.874 

58.269 

3.825 


MilestoneBscheduleachieved_Fractional_Year 

-35.031 

62.629 

-.559 


ContractEACLessInitialContractPrice 

-1.350 

.219 

-6.172 

Dependent Variable: CurrentLessInitialProgramCost 

= .491 

N = 816 


This initial regression of program eost growth, ineluding the independent variable 
for eontraet priee growth, had an unexpected negative coefficient for the variable 
ContractEACLessInitialContractPrice. A negative coefficient would indicate that as 
contract price growth increases, program cost growth decreases. Eollowing additional 
analysis, we obtained the result for the final regression shown in Table 13. 
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Table 13. Final Regression of Program Cost Growth (Current Estimate - Initial 
Cost), Including the Independent Variable for Contract Price Growth 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

17451.266 

2703.592 

6.455 


PROD_BasicContractType_FP_d 

-774.510 

895.329 

-.865 


RDTE_BasicContractType_FP_d 

1627.430 

1481.617 

1.098 


EMDBCpointdate_d 

-2961.289 

885.129 

-3.346 


CPIF_D 

578.116 

1846.206 

.313 


FFP_D 

1606.234 

2039.100 

.788 


FPIF_D 

-1001.003 

1904.692 

-.526 


CPAF_D 

-1127.917 

1872.168 

-.602 


CPFF_D 

-2009.671 

1901.696 

-1.057 


AcqType_Mod_D 

1591.428 

927.534 

1.716 


AcqSegment_C4ISR_D 

-16412.003 

1844.091 

-8.900 


AcqSegment_Missiles_D 

-15970.579 

1799.126 

-8.877 


AcqSegment_Aircraft_D 

-16507.851 

1732.202 

-9.530 


Program_Fractional_Year 

170.069 

77.163 

2.204 


ContractEACLessCurrentContractPrice 

.673 

1.348 

.499 


Contract Schedule Variance 

-55.089 

20.052 

-2.747 

Dependent Variable: CurrentLessInitialProgramCost 

= .565 

N = 409 


This final regression, which includes Contract Schedule Variance and switches 
from CurrentEACLessInitialContractPrice to CurrentEACLessCurrentContractPrice 
shows that as expected contract price overruns (ContractrEACLessCurrentContractPrice) 
increase, program cost growth increases. This logic is shown visually in Figure 14. The 
variable is not statistically significant, but was retained in this exploratory analysis The 
sign of the contract schedule variance for the current year, was negative, which was 
expected since negative contract variances indicate poor outcomes. Contract Schedule 
Variance was also significant; thus a poor contract schedule variance outcome was 
significant and, therefore, explained significant variation in program cost growth. In 
other words, an increase in Contract Schedule Variance (a good outcome) can be 
expected to reduce the final cost of a program (also a good outcome). The fact that the 
inclusion of Contract Schedule Variance caused the amended contract price variable to 
switch signs, and lose significance indicates that the correlations among the contract level 
variables is sufficiently complex that additional work on the model specification is 
required. 
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Figure 14. Typical Program and Contract Cost Growth Over Time 

Figure 14, a notional figure, shows contract EAC growth summing to the growth 
in the program SAR current estimate. This ignores the fact that programs contain 
significant government direct expenditures that do not flow through SAR reportable 
contracts. These government direct expenditures come from DoD work that 
complements the contracted efforts, but is not listed in the SAR. The regression in Table 
13 follows the expected relationship shown in Figure 14 between contract price growth 
and program cost growth. 

2. Regressions of Program and Contract Variances 

We focused this portion of our modeling efforts on relating cost, schedule, and 
engineering variances at the program level and earned value cost and schedule variances 
at the contract level. To thoroughly examine the effect of variables at both the program 
and the contract level, we ran regressions in an iterative manner, building to combined 
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program and contract variance regressions. First, however, we display a correlation table 
that contains the simple correlation coefficients among the program and contract 
variances. 

One is reminded that negative contract cost and schedule varianees refleet a bad 
outcome and negative program variances for cost and schedule reflect good outcomes. A 
positive sign in engineering variance is acceptable if there is a valid rise in the military 
requirements during the program, which EV captures. The eorrelations between program 
and contract variances are shown in Table 14 and significant correlations are highlighted. 
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Table 14. Program and Contract Variance Correlations 


Correlations 



Program 

Schedule 

Variances 

Program 

Engineering 

Variances 

Program Cost 

Variance 

Contract Cost 

Variance 

Contract 

Schedule 

Variance 

Program 

Pearson 

1 

-.008 

.245** 

-.241** 

-.280** 

Schedule 

Correlation 






Variances 

SIg. (2-talled) 


.801 

.000 

.000 

.000 


N 

904 

904 

904 

441 

441 

Program 

Pearson 

-.008 

1 

.301** 

-.147** 

.071 

Engineerin 

Correlation 






g 

Sig. (2-tailed) 

.801 


.000 

.002 

.137 

Variances 

N 

904 

904 

904 

441 

441 

Program 

Pearson 

.245** 

.301** 

1 

-.107* 

-.090 

Cost 

Correlation 






Variance 

Sig. (2-tailed) 

.000 

.000 


.024 

.058 


N 

904 

904 

904 

441 

441 

Contract 

Pearson 

-.241** 

-.147** 

-.107* 

1 

.224** 

Cost 

Correlation 






Variance 

Sig. (2-tailed) 

.000 

.002 

.024 


.000 


N 

441 

441 

441 

441 

441 

Contract 

Pearson 

-.280** 

.071 

-.090 

.224** 

1 

Schedule 

Correlation 






Variance 

Sig. (2-tailed) 

.000 

.137 

.058 

.000 



N 

441 

441 

441 

441 

441 


**. Correlation is significant at the 0.01 level (2-tailed). 
*. Correlation is significant at the 0.05 level (2-tailed). 


The correlation table captures the size and significance of the between each of the 
variances under investigation. Of particular interest are the significant correlations 
between program level variances and contract level variances. Program Schedule 
Variance is significantly related to both Contract Cost Variance and Contract Schedule 
Variance. Program Engineering Variance is significantly associated with Contract Cost 
Variance, and Program Cost Variance is significantly associated with Contract Cost 
Variance. While this table indicates that there are connections among key variances. 
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what is needed are models that isolate the variance connections, holding other variables 
constant. Therefore, regression analysis is required. Attention will be focused on 
regression equations in which relevant cost variance is the dependent variable. Other 
models might be considered in which Program Schedule Variance, Program Engineering 
Variance, and Contract Schedule Variance are dependent variables, but this is beyond the 
scope of this research and awaits further analysis. 

First, a program variances-only regressions were constructed, followed by 
contract earned value variances regression. These variance regressions were used to 
inform the selection of control variables for our combined program and contract 
variances regression. The intent of the last regression was to establish if there is a direct 
relationship between the variance metrics used to track programs and the variance metrics 
used to track contract performance. 

The first regression examined was for Program Cost Variance. Program Cost 
Variance is composed of multiple components. This research focused in part on the 
relationship of Program Cost Variance to Program Schedule (SV) and Program 
Engineering Variance (EV). Since cost variance is composed of SV, EV and additional 
categories of cost variance, it is inappropriate to use SV and EV as independent variables 
and with cost variance as the dependent variable. Doing so could have likely inflated the 
correlation of SV and EV to cost variance. To resolve this concern, the program variance 
regressions use cost variance less schedule and engineering cost variances. To simplify 
the discussion, this is designated as Program Net Cost Variance in the regressions 
employing this variable. 

It should also be noted that all of the variance measure are those reported in the 
SAR reports for the current period, which is frequently a calendar year. The information 
on cumulative variance through the period is not employed. Also, all Program Variance 
data are measured in FY10 $M. 

The initial regression in which Program Net Cost Variance is the dependent 
variable, regressed on the explanatory variables, is shown in Table 15. The acquisition 
segment dummy variables, the modification dummy variable, the EMD period dummy 
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variable, the year in which MS B was achieved, and the time since program establishment 
were included to isolate the effect of SV and EV on Program Net Cost Variance. 
Contract type dummy variables and EMD contract type dummy variables were also used 
to control for contract type effects. Each contract type dummy variable had an EMD 
contract type dummy variable associated with it, except for EPIE, which was never the 
predominant contract type an EMD contract. The EMD contract type dummy variables 
combined with contract type variables not tied to EMD estimates the effect of contract 
type during EMD. Therefore, the effect of a contract type dummy variable, which is not 
identified with EMD, should, when EMD is 0, identify the effect of predominant contract 
type during Production. 


Table 15. Initial Regression of Program Cost Variances 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

-34602.05 

38478.021 

-0.899 


AcqSegment_Aircraft_D 

-307.233 

268.769 

-1.143 


AcqSegment_C4ISR_D 

-320.491 

282.355 

-1.135 


Acq Seg m e nt_M i ss i I es_D 

-299.473 

269.202 

-1.112 


AcqType_Mod_D 

239.014 

175.469 

1.362 


EMDBCpointdate_d 

-282.386 

527.93 

-0.535 


MilestoneBscheduleachieved_Fractional_Year 

17.612 

19.197 

0.917 


P rog ram_F ract i o n al_Year 

3.425 

18.647 

0.184 


CPIF EMD CaseContractType D 

31.929 

605.822 

0.053 


FFP EMD CaseContractType D 

82.528 

800.656 

0.103 


CPAF_EMD_CaseContractType_D 

-217.799 

597.086 

-0.365 


CPFF_EMD_CaseContractType_D 

-478.406 

731.539 

-0.654 


CPIF_D 

-70.673 

319.889 

-0.221 


FFP_D 

-245.879 

288.641 

-0.852 


FPIF_D 

-579.157 

336.471 

-1.721 


CPAF_D 

-203.669 

325.312 

-0.626 


CPFF_D 

508.229 

346.098 

1.468 


Program Schedule Variance 

1.13 

0.307 

3.679 


Program Engineering Variance 

1.059 

0.202 

5.235 

Dependent Variable: Program Net Cost Variance, Constant2010$M 

= .086 

N = 902 


The very low t statistics found for the Contract Type EMD Case Contract Type 
dummy variables indicated that these variables should be eliminated unless inserted as a 
control variable from the final regression. Subsequent regressions (not shown in this 

research) also revealed that Program_Fractional_Year (the time since program initiation) 
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had a non-significant t statistic; thus, Prograni_Fractional_Year was removed. The 
variables in the final program cost variance regression shown in Table 16 are otherwise 
the same as those shown in the initial program cost variance regression. 


Table 16. Final Regression of Program Cost Varianees 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

-28472.465 

29499.223 

-0.965 


AcqSegment_Aircraft_D 

-342.875 

262.09 

-1.308 


AcqSegment_C4ISR_D 

-330.648 

277.858 

-1.19 


Acq Seg m e nt_M i ss i I es_D 

-328.202 

264.874 

-1.239 


AcqType_Mod_D 

268.756 

171.754 

1.565 


EMDBCpointdate d 

-420.883 

180.665 

-2.33 


MilestoneBscheduleachieved_Fractional_Year 

14.591 

14.761 

0.988 


CPIF_D 

-78.577 

273.31 

-0.288 


FFP_D 

-281.199 

252.884 

-1.112 


FPIF_D 

-625.172 

305.184 

-2.049 


CPAF_D 

-281.118 

267.87 

-1.049 


CPFF_D 

404.442 

298.584 

1.355 


Program Schedule Variance 

1.13 

0.306 

3.687 


Program Engineering Variance 

1.064 

0.198 

5.378 

Dependent Variable: Program Net Cost Variance, Constant2010$M 

= .084 

N = 902 


In the final model, the dropped slightly, but the remaining t statistics improved 
enough to justify eliminating the EMD contract-type dummy variables. The program 
segment variables are not statistieally different from the reference Ships binary variables. 
The Milestone B variable is retained for future consideration in a re-specified model, and 
only the FPIF binary variables is significantly lower than the reference hybrid contract 
variable. Notiee that the largest negative effeet oecurred when FPIF was the predominant 
eontraet type. However, because FPIF was never a predominant contact type during 
EMD, this result cannot be applied to a hypothetical situation in which an FPIF contract 
was the predominant contract type during the EMD phase. The positively significant 
effect of Program Schedule Variance and Program Engineering Variance on Program Net 
Cost Varianee (Program Cost Varianee less Sehedule and Engineering Variance) is an 
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important finding. The coefficients of these two variables are close to 1, so that, other 
things equal a one dollar increase in each of these variables increases Program Net Cost 
Variance by about one dollar. 

The next regression examined was for contract earned value variances. This 
regression sought to better understand the effect of Contract Schedule Variance on 
Contract Cost Variance. The initial regression controls for contract type using contract 
type dummy variables, including contract dummy variables designed to estimate the 
effect found during EMD. The initial regression of Contract Cost variances on the 
selected explanatory variables is shown in Table 17. 


Table 17. Initial Regression of Contract Variances 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

0.531 

9.482 

0.056 


CPIF_EMD_CaseContractType_D 

3.014 

9.549 

0.316 


F F P_E M D_CaseContractT ype_D 

-0.874 

46.412 

-0.019 


CPAF_EMD_CaseContractType_D 

-4.001 

8.481 

-0.472 


C P FF_E M D_CaseContractT y pe_D 

3.685 

16.496 

0.223 


CPIF_D 

-7.03 

10.593 

-0.664 


FFP_D 

0.343 

13.27 

0.026 


FPIF_D 

4.201 

10.933 

0.384 


CPAF_D 

-7.461 

10.526 

-0.709 


CPFF_D 

-7.733 

11.499 

-0.672 


Contract Schedule Variance 

0.797 

0.164 

4.867 

Dependent Variable: Contract Cost Variance 

= .061 

N = 441 


We find that increases in Contract Schedule Variance have a significant positive 
effect on Contract Cost Variance. Contract type, whether interacting or not interacting 
with EMD, does not have a significant effect on Contract Cost Variance relative to the 
reference contract. 

After examining multiple iterations of the model, the final regression of Contract 
Cost Variance is examined. This revised model eliminated the dummy variables in which 
EMD is tied contract type because of poor t-statistics. After revising the initial 
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hypothesis and specifying that acquisition segment dummy variables, the modification 
acquisition type, date at which MS B was achieved, and the years since MDAP inception 
belong in the model, we obtain the results shown in Table 18. 


Table 18. Final Regression of Contract Cost Variance 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

-1141.612 

1444.43 

-0.79 


AcqSegment_Aircraft_D 

14.983 

8.857 

1.692 


AcqSegment_C4ISR_D 

26.301 

9.484 

2.773 


AcqSegment_Missiles_D 

24.144 

9.246 

2.611 


AcqType_Mod_D 

8.303 

7.214 

1.151 


MilestoneBscheduleachieved_Fractional_Year 

0.555 

0.721 

0.769 


Program_Fractional_Year 

1.142 

0.678 

1.685 


CPIF_D 

-2.89 

10.417 

-0.277 


FFP_D 

4.183 

13.492 

0.31 


FPIF_D 

11.941 

11.369 

1.05 


CPAF_D 

-4.963 

10.442 

-0.475 


CPFF_D 

0.437 

11.376 

0.038 


Contract Schedule Variance 

0.733 

0.163 

4.495 

Dependent Variable: Contract Cost Variance 

= .086 

N = 441 


Many of the additional variables included are statistically significant. However, 
the predominant contract type variables continue to be statistically non-significant 
compared with the reference hybrid contract. Importantly, the significant positive 
relationship between Contract Schedule Variance and Earned Value Cost Variance 
continues to apply. This means that an increase in Budgeted Cost of Work Performed 
less Budgeted Cost of Work Scheduled (a positive outcome) is associated with an 
increase in Budgeted Cost of Work Performed less Actual Cost of Work Performed (also 
a positive outcome). 

After conducting program variance-only and contract variance-only regressions, 
the following regressions shown in Table 19 with both, program and contract variances 
were examined. The initial regression sought to establish the effect of the Contract Cost 
Variance, Contract Schedule Variance, Program Engineering Variance and Program 
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Schedule Variance on Program Net Cost Variance. Additional variables were added to 
control for the effects of being in EMD and the predominant contract type by program 
during EMD. 


Table 19. Initial Regression of Program and Contract Variances 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

-38.77 


-0.273 


Program Engineering Variance 
Constant2010$M 

1.084 

0.186 

3.997 


Program Schedule Variance 

Constant2010$M 

1.729 

0.232 

4.771 


EMDBCpointdate_d 

-297.978 

-0.071 

-1.463 


EMD_CPIF_D 

692.664 

0.169 

3.064 


EMD_CPAF_D 

96.755 

0.027 

0.486 


EMD_CPFF_D 

405.657 

0.08 

1.579 


Contract Cost Variance 

0.577 

0.016 

0.327 


Contract Schedule Variance 

-3.517 

-0.028 

-0.572 

Dependent Variable: Program Net Cost Variance Constant2010$M 

R^ = .104 

N = 439 


With the Program Engineering and Schedule Variances included in the regression, 
the Contract Cost and Contract Schedule Variances are not significant. We can note from 
Table 14 that the eorrelation between Program Sehedule Variance and both Contract Cost 
Variance and Contract Schedule Variance are statistically significant, and the correlation 
between Program Engineering Variance and Contract Cost Variance are statistically 
significant. One can conjecture that multicollinearity might have impacted the 
signifieanee of Contraet Variance variables. In the respecification of the model, the 
Program Varianee explanatory variables are deleted, and additional variables are added to 
determine whether an alternative model might have superior statistical properties. The 
final regression of program and contract variances resulting from these changes is shown 
in Table 20. 
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Table 20. Final Regression of Program and Contract Variances 


Model 


Coefficient 

Std. Error 

t-statistic 


(Constant) 

31915.502 

58874.493 

0.542 


Acq Seg m e nt_Ai rcraf t_D 

150.025 

365.233 

0.411 


AcqSegment_C4ISR_D 

-296.542 

391.406 

-0.758 


AcqSegment_Missiles_D 

-249.867 

375.703 

-0.665 


AcqType_Mod_D 

-29.488 

304.061 

-0.097 


EMDBCpointdate_d 

71.881 

827.471 

0.087 


MilestoneBscheduleachieved_Fractional_Year 

-15.458 

29.383 

-0.526 


Program_Fractional_Year 

-63.91 

30.814 

-2.074 


C PI F_E M D_CaseContractT ype_D 

-743.446 

894.203 

-0.831 


FFP_EMD_CaseContractType_D 

1244.68 

2049.642 

0.607 


CPAF_EMD_CaseContractType_D 

-1061.793 

903.484 

-1.175 


CPFF_EMD_CaseContractType_D 

-1462.308 

1079.94 

-1.354 


CPIF_D 

132.394 

509.539 

0.26 


FFP_D 

34.286 

604.259 

0.057 


FPIF_D 

-238.723 

532.09 

-0.449 


CPAF_D 

68.264 

520.106 

0.131 


CPFF_D 

517.904 

541.03 

0.957 


Contract Cost Variance 

-3.281 

1.939 

-1.692 


Contract Schedule Variance 

-10.073 

6.722 

-1.499 

Dependent Variable: SubtotalCurrentChangesConstant2010$M 

= .062 

N = 439 


The newly specified regression, which now includes (total) Program Cost 
Variance, significantly enhances the statistical significance of the Contract Variances and 
yields coefficient signs that are expected; increasing Contract Cost Variance and Contract 
Schedule Variance correlated with decreasing Program Cost Variances. This negative 
correlation actually indicates that contract and program variances are traveling in the 
same direction, since the sign of a good outcome for contracts is positive, while the sign 
of a good outcome for programs is negative. Therefore, a connection between program 
level data and contract level data has been identified. In addition. Program Fractional 
Year is a statistically significant variable in this revised model. 

The understanding of program and contract variances we built using multiple 
regression analysis supports our model of the relationship between contract price growth 
and program cost growth, shown in Figure 15. 
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Figure 15. Relationship Between Variables 

The relationships shown in Figure 15 could form the starting point for further 
analysis of the relationship between contract price growth and program cost growth. 
Chapter VI includes suggestions regarding future analysis. Of particular interest is 
structural equation modeling, which offers the possibility to separate direct and indirect 
effects between variables effecting both contract price growth and program cost growth. 
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VI. SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 


A. DISCUSSION OF RESULTS 

This research focused on quantifying relationships between acquisition program 
attributes, choices, and outcomes. The results recorded in Chapter V, while exploratory 
in nature, show the relationships that we found in attempting to address each of our 
research questions. In this chapter, we review those results, provide limitations of the 
study, and present recommendations for further research. 

Our results do not provide a definitive conclusion to all our research questions, 
but they do offer insight toward the relationships between program and contract 
variances, along with contract types. As discussed in this study, one area of potential 
concern is that the use of fixed-price contracts during R&D, which may limit contractors’ 
efforts to identify possible technical risks early in a program that would prevent future 
system design problems. There were only 31 fixed-price contracts started during EMD, 
and no program had predominant fixed-price contracts during the EMD phase. The three 
programs with predominantly fixed-price contracts initiated under RDT&E 
appropriations performed unfavorably on program costs compared to other contract types 
in the dataset in this exploratory analysis. This may be attributed to the establishment of 
a fixed-price contract amount. The CV, SV, and EV all performed worse for programs 
with fixed-price RDT&E, when compared to the remaining MDAPs. This could be a 
result of the increased uncertainty and technical risk experienced during RDT&E. 
Programs with a majority of contract spending on fixed-price contracts during the 
development phase may not be able to appropriately hedge technical risk. This might be 
because the ceiling price could provide incentives for a contract change to be approved. 
Although not conclusive, the use of incentives and award fees may run higher costs and 
longer schedules, but could provide prevention of expensive future problem 
identification. 

When comparing the different acquisition segment results, C4ISR performed the 
least favorably, with the highest schedule growth of 13% and the second highest cost 
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growth of 10%. The aircraft segment had the highest cost growth at just over 10%, but 
the lowest schedule growth at 1.5%. The missiles, weapons, and ammunition segment 
fell into the middle of the three categories; the single submarine program we analyzed 
was not comparable to the other segments. 

Our findings regarding CV, SV, and EV as leading indicators of later period CV, 
SV, and EV were only slightly conclusive, in part due to SV and EV not being 
extensively studied as dependent variables. Our findings regarding CV were partially 
conclusive. We found that an increase in program year (the number of years since 
program inception at MS B) was significantly positively correlated with an increase in 
contract price growth (table 11) and program cost growth (tables 9 and 13). Also, an 
increase in program year was significantly negatively correlated with an increase in 
program cost variance (table 20). Program cost growth is a measure of scope increases 
and overruns, while program cost variance is a measure of changes during the past year to 
a program’s expected cost. The fact that the program year independent variable has 
opposite and significant signs for the two measures of program cost changes likely 
indicates that rebaselining significantly impacts the relationship of one or both to 
program year. Program year has a significant t statistic for the final program variance 
regression (table 20) than contract cost variance or contract schedule variance. The 
strength of this relationship and the opposite effect for program growth and program 
variance measures could indicate that this variable is actually capturing the effect of 
rebaselining on program metrics. Rebaselining could be a significant source of the 
limited correlation between contract variances and program variances in the program 
variances regression. Eurther study of the relationship between program variances and 
further study of the effects of rebaselining could improve the explanatory power of our 
models. 

The results achieved using regressions established preliminary findings. Aircraft, 
C4ISR, and Missile segments all showed significantly lower cost growth than Ships. 
Program managers of ships and submarines should be cognizant of the risk of higher cost 
growth that exists on already expensive programs. Additionally, regressions showed 
modifications actually ran higher cost growth than did new starts and initial programs. 
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Although not at a significant level, the predominant use of fixed-price contracts during 
production was associated with lower cost growth. This follows our discussion of risk 
with regards to contraet type in Chapter II. 

We are one of the first to research the relationship of Earned Value Management 
contract variances and program variances. The connection between program and contract 
variances identified supports the view that contract price growth affects program cost 
growth. Understanding exaetly how program growth is affected by contract growth will 
allow program managers to better eontrol eosts. No conclusive effeets were established, 
but this research has laid the ground work for future research such as SEM discussed in 
Section D to refine the results achieved. 

B. RECOMMENDATIONS 

In this research, we partially confirmed the reservations of acquisition 
professionals who believe the push for fixed-price R&D from sources such as Under 
Seeretary Carter’s Better Buying Power may not always be the best contract type. We 
also determined that programs with a majority of fixed-price RDT&E eontraets tend to 
have lower cost growth overall and lower cost growth attributable to schedule growth, but 
higher engineering cost growth compared to all MDAPs. By treating engineering cost 
growth as a proxy for technical changes, and by eonsidering the attendant technical risks 
to those ehanges, it is elear that fixed-price RDT&E saves money and time, but increases 
teehnieal risk. This increased technical risk may be acceptable for RDT&E spending 
during the production phase, but would likely be unadvisable during EMD, when the 
costs of discovering problems in systems engineering are substantial. 

We examined the use of fixed-price eontraets for RDT&E spending in this 
researeh, but we did not examine the use of fixed-price eontraets during EMD beeause 
none of the MDAPs in our sample had fixed-price contracts as the majority contract type 
for EMD. However, it is possible that fixed-price contracts may be successfully extended 
into EMD for a limited number of systems, partieularly those that exceed the now- 
strengthened requirements for minimum TRLs. Program managers should proceed with 
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caution, due to the potential for increased technical risk. Only 9% of RDT&E contracts 
in our sample were fixed-price, and the vast majority of those were during the production 
phase. 


C. STUDY LIMITATIONS 

The data we used in this study came directly from each program’s SAR, including 
cost variances at the program level and earned value at the contract level. We analyzed a 
relatively small number of programs (31) due to the limited availability of data that fit 
our study’s requirements and to the labor-intensive nature of the collection. The dataset 
included multiple years and contracts for each program, which provided us with a 
significant number of observations. 

SAR data have many limitations that we observed during this study. Although 
recent changes to regulations driven by WSARA were made in an attempt to reduce 
rebaselining, our period of study went back as far as 1997 for some programs. Due to 
shifting rebaselining policy over time, it was difficult to determine the decisions made 
and the effects of these decisions on variances after a program was rebaselined. After 
reviewing multiple SAR variances at the program and contract level, there were missing 
explanations as to the cause of variances required. Multiple SARs had vague information 
regarding contract type. It appeared that cost variances were allocated inconsistently 
between SAR categories and between years. The cost information provided for future 
years reflected projected budget values and was not always consistent with cost 
estimates. Multiple contract types were listed with no identification of dollar amounts or 
designation of dominant type. 

Even with the limitations of the SARs, we were able to build an extensive dataset. 
The data also had their own limitations. The low number of fixed-price contracts utilized 
during the EMD phase prevented us from producing thorough results of the effects of 
fixed-price development contracts on program variances. Due to a lack of variability in 
predominant contract type during the EMD phase and due to no program having a 
predominant fixed-price contract type, we removed those programs with predominant 
RDT&E contract type to compare differences in production contract type. Also, as we 
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have indicated, the regression analysis must be viewed as exploratory. The models 
estimated are a step on the road to hypothesizing causal models that may succeed in 
obtaining definitive findings. 

D. AREAS FOR FUTURE RESEARCH 

1. Structural Equation Modeling 

In addition to looking at definable, quantitative cost growth, it would also be 
helpful to examine root sources of variance within programs. In Chapter II we discussed 
risk in different acquisition phases. Risk in acquisition programs is an undefined 
characteristic that program managers continually try to control. Programs have 
designated risk reserves, but these reserves and levels of uncertainty or risk are not 
reportable on SARs. Although risks are understood in the acquisition community, they 
could have direct effects on program or contract variances. 

Program risk can be characterized by three subparts: cost, schedule, and technical 
risk. Once these risks are recognized, the result could include a change in price, 
schedule, or technology that could ultimately correspond to a program CV, SV, or EV. 
Any single change could lead to changes throughout a program. For example, funding 
from Congress could be reduced, affecting the quantity of systems affordable, adjusting 
the design, and lengthening the schedule. 

To allow for more advanced statistical methods of cross-sectional, time-series 
analysis, we recommend use of structural equation modeling (SEM) to analyze program 
and contract variance data. A thorough investigation should be conducted, including an 
investigation of the use of exploratory factor analysis to identify latent variables, and the 
results of this investigation should be used to assess the applicability of SEM to program 
and contract variance data. SEM differs from the prior techniques in its “ability to 
simultaneously estimate multiple dependence relationships ... while also incorporating 
multiple measures for each [construct]” (Hair et ak, 2009, p. 609). Constructs would be 
the dependent and independent variables included in the analysis. SEM examines both 
interdependent and dependent relationships simultaneously in a manner akin to 
combining factor analysis and multiple regression analysis (Hair et ak, 2009). 
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SEM models also permit the simultaneous variance of multiple factors including 
risk. This method allows the model to include conceptually constructed variables that 
underlie the measurable variables and could help explain the full effect of fixed-price 
incentive R&D contracts within MDAPs on later period cost and schedule variance 
during the production and deployment phase. The use of SEM could allow for the 
examination of variance of multiple, interrelated, simultaneously varying factors such as 
the cost, schedule, and engineering variances of a program and the earned value cost and 
schedule variances of that program’s contracts. At the same time, it incorporates 
unobserved variables that could be categorized as programmatic risk and its various risk 
components. These risk measures would mediate the relationship between contract type 
and other variables and the programs’ cost objectives, and thereby clarify the effect of 
contract type on the desired outcome. The objective of this future research would be to 
determine the effects of the increased usage of fixed-price R&D contracts during EMD 
on long-term program CV, SV, and EV. It would also look to determine whether 
preproduction CV, SV, and EV and the associated contract variance measures are 
indicators of CV, SV, and/or EV and contract performance during production. 

2. Normality 

Unlike regression analysis, normality of the variables is a more important 
assumption when SEM is employed. Therefore, another area deserving future research is 
the source of non-normality in program cost variance. This topic would lend itself to a 
research based on frank discussions with past program managers about information¬ 
hiding and moral hazard in the principal-agent relationship between Congress and 
program managers. It would also lend itself to further exploration of the characteristics 
of programs that exhibit drastic deviations from normality. 
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F. 


SUMMARY 


This research sought to improve understanding of the interrelationship between 
program and contract metrics by quantify the relationships between program and contract 
outcomes. Our exploratory efforts to explain program cost increases using program and 
contract outcomes resulted in the explanation of 56.6 percent of program cost increases. 
Our efforts to explain program cost variance indicated that only 6.2 percent of program 
cost variance could be explained. As indicated above, however, a model of the small 
portion of the total variance explained when analyzing a response to a policy change, 
such as a change in contract type, provided the policy change variable is statistically 
significant. Nevertheless, additional modeling beyond the exploratory effort undertaken 
here could increase the ability of both models to explain overall program outcomes, and 
how these change when a particular policy variable changes. This research also 
attempted to show how preproduction CV, SV, and EV data within acquisition programs 
effect future variances. The results were not significantly quantifiable, and additional 
research is recommended. Our results indicated several possible areas of future research, 
including the non-normality in SAR measures and SEM. Both areas offer the possibility 
of increasing the effectiveness of the models explored in this project. 
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APPENDIX A. LIST OF VARIABLE FOR ANALYSIS 


Variables used in the research write-up 

Name 

Explanation 

AcqSegment_Aircraft_D 

1, if Acquisition Segment is aircraft; 0, otherwise 

AcqSegment_C4ISR_D 

1, if Acquisition Segment is C4ISR; 0, otherwise 

AcqSegment_Missiles_D 

1, if Acquisition Segment is missiles; 0, otherwise 

AcqSegment_Ships_D 

1, if Acquisition Segment is ships; 0, otherwise 

AcqType_Mod_D 

1, if program Acquisition Type is a modification; 0, otherwise 

AcqType_New_D 

1, if program Acquisition Type is a new start; 0, otherwise 

CECostVarianceConstant2010$M 

Current Program Cost Estimate through the end of Production (in 
2010 Constant Millions of Dollars); sum of prior and current cost 
estimate changes and baseline cost estimate 

CONTOTH_D 

1, if contract type is Other; 0, otherwise 

CONTOTH_EMD_CaseContractTy 

pe_D 

COMPUTE 

CONTOTH_EMD_CaseContractType_D=EMDBCpointdate_d * 
CONTOTH_D 

Contract_BasicContractType_CP_ 

D 

1, if CP is Basic Contract Type for an individual contract; 0, 
otherwise (FP, fixed price; or CP, cost plus) 

Contract_BasicContractType_FP_ 

D 

1, if FP is Basic Contract Type for an individual contract; 0, 
otherwise (FP, fixed price; or CP, cost plus) 

ContractEACLessCurrentContract 

Price 

COMPUTE 

ContractEACLessCurrentContractPrice=ProgramManagerEstimated 
PriceAtCompletion$M - TargetCurrentContractPrice$M 

ContractEACLessInitialContractPri 

ce 

COMPUTE 

ContractEACLesslnitialContractPrice=ProgramManagerEstimatedPri 
ceAtCompletion$M - TargetlnitialContractPrice$M 

ContractID 

Contract ID associated with a contract at the program level (NOTE: 
this contract ID repeats for each program; e.g. each program has a 
contract "1") 

ContractName 

Contract Name 

ContractNumber 

Contract Number (obtained from SAR and corrected, as necessary, 
using FPDS) 

Contractor 

Contractor Name 

ContractType 

Contract Type from SAR, adjusted using FPDS data 

Contract Cost Variance 

Current Cost Variances (in Millions of Dollars) 

Contract Schedule Variance 

Current Schedule Variances (in Millions of Dollars) 

CPAF_D 

1, if contract type is CPAF; 0, otherwise 

CPAF_EMD_CaseContractType_D 

COMPUTE CPAF_EMD_CaseContractType_D=EMDBCpointdate_d * 
CPAF_D 

CPFF_D 

1, if contract type is CPFF; 0, otherwise 

CPFF_EMD_CaseContractType_D 

COMPUTE CPFF_EMD_CaseContractType_D=EMDBCpointdate_d * 
CPFF_D 

CPIF_D 

1, if contract type is CPIF; 0, otherwise 
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CPIF_EMD_CaseContractType_D 

COMPUTE CPIF_EMD_CaseContractType_D=EMDBCpointdate_d * 
CPIF_D 

CurrentLessInitialContractPrice 

COMPUTE 

CurrentLesslnitialContractPrice=TargetCurrentContractPrice$M - 
TargetlnitialContractPrice$M 

CurrentLessInitialProgramCost 

COMPUTE 

CurrentLesslnitialProgramCost=CECostVarianceConstant2010$M - 
SARBaselineProdEstConstant2010$M 

EMD_BasicContractType_CP_D 

1, if CP Predominant contract type during EMD; 0, otherwise (CP, 

Cost Plus; or FP, Fixed Price) 

EMD_BasicContractType_FP_D 

1, if FP Predominant contract type during EMD; 0, otherwise (CP, 

Cost Plus; or FP, Fixed Price) 

EMD_CONTOTH_D 

1, if Predominant contract type during EMD = Flybrid/Other; 0, 
otherwise 

EMD_CPAF_D 

1, if Predominant contract type during EMD = CPAF; 0, otherwise 

EMD_CPFF_D 

1, if Predominant contract type during EMD = CPFF; 0, otherwise 

EMD_CPIF_D 

1, if Predominant contract type during EMD = CPIF; 0, otherwise 

EMD_FFP_D 

1, if Predominant contract type during EMD = FFP; 0, otherwise 

EMD_FPIF_D 

1, if Predominant contract type during EMD = FPIF; 0, otherwise 

EMD_D 

1, if data represents EMD period. 0, if data represents Production 
period 

EMDpredominantBasicContractT 

Predominant contract type during EMD (CP, Cost Plus; or FP, Fixed 

ype 

Price) 

FFP_D 

1, if contract type is FFP; 0, otherwise 

FFP_EMD_CaseContractType_D 

COMPUTE FFP_EMD_CaseContractType_D=EMDBCpointdate_d * 
FFP_D 

FPIF_D 

1, if contract type is FPIF; 0, otherwise 

FPIF_EMD_CaseContractType_D 

COMPUTE FPIF_EMD_CaseContractType_D=EMDBCpointdate_d * 
FPIF_D 

MilestoneBscheduleachieved_Fra 

Sum of MilestoneBscheduleachieved_Year + 
MilestoneBscheduleachieved_Month_12 (NOTE: December 2010 is 

ctional_Year 

then represented as 2011.00) 

PROC_D 

1, if appropriation is Procurement; 0, otherwise 

PROD_BasicContractType_CP_D 

1, if Cost-plus; 0, otherwise 

PROD_BasicContractType_FP_D 

1, if Fixed-price; 0, otherwise 

PROD_CONTOTH_D 

1, if Flybrid/Other; 0, otherwise 

PROD_CPAF_D 

1, if CPAF; 0, otherwise 

PROD_CPFF_D 

1, if CPFF; 0, otherwise 

PROD_CPIF_D 

1, if CPIF; 0, otherwise 

PROD_FFP_D 

1, if FFP; 0, otherwise 

PROD_FPIF_D 

1, if FPIF; 0, otherwise 

PRODpredominantBasicContractT 

Predominant basic contract type during Production (Fixed-price, FP; 

ype 

or cost-plus, CP) 

PRODpredominantContractType 

Predominant contract type during Production (FFP, FPIF, CPIF, 
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CPAF, CPFF, Hybrid/Other) 

Progrann_Fractional_Year 

Years since program initiation 

ProgramandContractlD 

A unique Contract ID associated with a contract representing a 
single scope of work. To accommodate multiple programs and 
contracts, the ID is structured with a leading 1, followed by the 
Program ID followed by a leading 1, followed by the Contract 

ID For greater than 99 programs or contracts, a leading 2 

would be utilized. 

Program Engineering Variance 
Constant2010$M 

Current period Cost Estimate Changes from the Baseline due to 
Engineering factors (in 2010 Constant Millions of Dollars) 

ProgramlD 

ID numbers corresponding to each program 

ProgramManagerEstimatedPrice 

AtCompletion$M 

Program Manager's Estimated Price at Completion (in Millions of 
Dollars) 

PROG RAM NAME 

The name generally associated with the program in recent SARs 

Program Net Cost Variance 
Constant2010$M 

SubtotalCurrentChangesBY$M - Program Schedule Variance BY$M 
- Program Engineering Variance BY$M. Commonly referred to as 
Program Cost Variance (or just Cost Variance) in research. 

Program Schedule Variance 
Constant2010$M 

Current period Cost Estimate Changes from the Baseline due to 
Schedule factors (in 2010 Constant Millions of Dollars) 

RDTE_BasicContractType_CP_D 

1, if FP Predominant contract type for RDT&E; 0, otherwise (CP, 

Cost Plus; or FP, Fixed Price) 

RDTE_BasicContractType_FP_D 

1, if CP Predominant contract type for RDT&E; 0, otherwise (CP, 

Cost Plus; or FP, Fixed Price) 

RDTE_CONTOTH_D 

1, if Predominant contract type for RDT&E = Hybrid/Other; 0, 
otherwise 

RDTE_CPAF_D 

1, if Predominant contract type for RDT&E = CPAF; 0, otherwise 

RDTE_CPFF_D 

1, if Predominant contract type for RDT&E = CPFF; 0, otherwise 

RDTE_CPIF_D 

1, if Predominant contract type for RDT&E = CPIF; 0, otherwise 

RDTE_D 

1, if appropriation is RDT&E; 0, otherwise 

RDTE_FFP_D 

1, if Predominant contract type for RDT&E = FFP; 0, otherwise 

RDTE_FPIF_D 

1, if Predominant contract type for RDT&E = FPIF; 0, otherwise 

RDTEpredominantBasicContractT 

ype 

Predominant contract type for RDT&E contracts (CP, Cost Plus; or 

FP, Fixed Price) 

SAR_Fractional_Year 

Sum of SAR_Year + SAR_Month_12 (NOTE: December 2010 is then 
represented as 2011.00) 

SARBaselineProdEstConstant201 

0$M 

SAR Baseline Cost Estimate through the end of Production (in 2010 
Constant Millions of Dollars) 

SubtotalCurrentChangesConstant 

2010$M 

Subtotal of Current period Cost Estimate Changes from the Baseline 
(in 2010 Constant Millions of Dollars) 

TargetCurrentContractPrice$M 

Current Target Contract Price (in Millions of Dollars) 

TargetlnitialContractPrice$M 

Initial Target Contract Price (in Millions of Dollars) 
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APPENDIX B. NORMALITY OF VARIANCE DATA 


For future analysis, particularly SEM, it may be helpful to understand the extent 
to which the variance data are normally distributed. Figure 17 and 18 are histograms of 
program schedule variance and contract schedule variance, demonstrating this kurtosis. 
Each histogram also shows the expected normal distribution of the data, given the sample 
standard deviation. When interpreting these figures it is useful to note that negative 
program variances represent better-than-expected performance, while negative contract 
variances represent poorer-than-expected performance. 



Mean = 16.93 
Std. Dev. = 140.379 
N = 904 


Figure 16. Program Schedule Variance Histogram With Expected Normal 

Distribution Overlaid 


105 








Schedule Variance - Net Change 


Mean = -1.8 
Std. Dev. = 13.391 
N = 441 


Figure 17. Contract Schedule Variance Histogram With Expected Normal 

Distribution Overlaid 


There are two possible explanations for this kurtosis. The first is a failure by 
program managers and contractors to report undesirable information due to improper 
incentives corrupting the flow of information from agent to principal at the program and 
contract levels. The second explanation is that the data represent two separate, overlaid 
normal distributions. One distribution would represent the typical behavior of programs: 
incremental failures and successes resulting in relatively small standard deviations and a 
more compact distribution of results. The other distribution would represent the aberrant 
behavior of programs: large over- or under-estimation in cost requirements, schedule 
length, technical maturation, or integration risk. These more unusual events would result 
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in larger standard deviations and a more “flat” distribution of results. It is likely that both 
explanations are partially responsible for the non-normality present. 

Program managers are incentivized to report variances that are advantageous to 
their program. This leads to a suppression of variances representing poorer-than- 
expected outcomes due to the subsequent increase in program risk increasing the 
possibility of program cancelation. Also, variances representing better-than-expected 
outcomes are suppressed due to the possibility of the program losing funding, thereby 
precluding technical risk reduction. Program managers are incentivized to think about 
what is best for their program and Service, while Congress has different goals. Absent 
perfect incentives, differing objectives result in the agent performing in a manner other 
than how the principle intended (Laffont & Martimort, 2002). Information can be 
suppressed by an agent (the program manager) to the extent that information asymmetry 
exists with the principal (Congress). Congress limits information asymmetry through 
statutory requirements. Program managers may not necessarily be dishonest in their 
reporting; willful blindness in the form of excessive pessimism or optimism may also be 
to blame for near-zero variances. Reporting fewer zero variances would increase the 
standard deviation of the sample and potentially lower the number of outliers. If outliers 
found in the data actually represent past variances suppressed to zero or near-zero values, 
then the standard deviation would not increase, but the amount of kurtosis would 
decrease. 

The second explanation, that the data represent two separate, overlaid normal 
distributions, is also likely to contribute to the kurtosis. If this is correct, two 
distributions are represented in the data. One distribution represents the typical behavior 
of programs—incremental failures and successes resulting in relatively small standard 
deviations—and another distribution represents the aberrant behavior of programs—large 
over- or under-estimation in cost requirements, schedule length, technical maturation, or 
integration risk, resulting in larger standard deviations. Causes for this type of large 
program or contract variance could be a large over- or underestimation of the effort 
required to complete a task, test event failures, poor estimating, or the realization of an 
accepted risk. Risks assumed by the government, such as integration risk, may play a 
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large part, particularly in dramatic program cost increases. It is likely that both possible 
causes of the kurtosis are partially responsible for the non-normality present in the data. 
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